Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.23 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2466285 2466236 2026-10-11T07:37:04Z Stashbot 7414 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2466285 wikitext text/x-wiki === 2026-10-11 === * 07:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-09 === * 18:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.1.3 ([[phab:T414595|T414595]]) * 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.1.3 ([[phab:T414595|T414595]]) === 2026-10-08 === * 17:01 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:58 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-07 === * 15:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T440326|T440326]]) * 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T440326|T440326]]) * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-06 === * 17:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2026-10-05 === * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T439237|T439237]]) * 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T439237|T439237]]) * 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T439237|T439237]]) * 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T439237|T439237]]) * 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 11:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-10-02 === * 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 13:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) === 2026-10-01 === * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 14:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-30 === * 14:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 09:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-29 === * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-25 === * 18:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:31 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-90.tools.eqiad1.wikimedia.cloud to the cluster * 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-87 * 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-87 * 13:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 13:45 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster * 13:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) * 13:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-115 ([[phab:T438255|T438255]]) * 13:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-115 ([[phab:T438255|T438255]]) * 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:57 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-89.tools.eqiad1.wikimedia.cloud to the cluster * 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 01:43 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster * 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) === 2026-09-24 === * 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 20:26 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster * 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) * 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 20:17 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-114.tools.eqiad1.wikimedia.cloud to the cluster * 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) * 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 20:05 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-89.tools.eqiad1.wikimedia.cloud to the cluster * 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:20 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-88.tools.eqiad1.wikimedia.cloud to the cluster * 19:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:38 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-87.tools.eqiad1.wikimedia.cloud to the cluster * 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:02 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-86.tools.eqiad1.wikimedia.cloud to the cluster * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 192 cores, 262144 ram ([[phab:T439157|T439157]]) * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 192 cores, 262144 ram ([[phab:T439157|T439157]]) * 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:38 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-85.tools.eqiad1.wikimedia.cloud to the cluster * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:22 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-84.tools.eqiad1.wikimedia.cloud to the cluster * 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-23 === * 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:11 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:59 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:57 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:56 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-22 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-backup-1.tools.eqiad1.wikimedia.cloud * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-backup-1.tools.eqiad1.wikimedia.cloud * 10:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools--k8s-backup-1.tools.eqiad1.wikimedia.cloud * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools--k8s-backup-1.tools.eqiad1.wikimedia.cloud * 07:51 godog: correction that's tools-k8s-worker-nfs-74 * 07:48 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 07:45 godog: hard reboot tools-k8s-worker-nfs-70 - ran out of memory * 07:36 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-21 === * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 10:15 godog: reboot tools-k8s-worker-nfs-70 * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:39 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:34 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:33 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:33 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:33 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:33 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 08:30 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:29 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:23 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:14 dcaro: rebooting bastion, connections will close/fail for a few seconds * 07:41 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 07:36 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2026-09-18 === * 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 14:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 13:39 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) === 2026-09-16 === * 13:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-16 * 12:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 * 12:33 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 12:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2026-09-15 === * 18:09 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-09-14 === * 22:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 22:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-11 === * 09:08 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:58 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-10 === * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 14:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-08 === * 14:40 komla@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T436614|T436614]]) * 14:19 komla@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T436614|T436614]]) * 14:13 komla@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T436614|T436614]]) * 14:13 komla@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T436614|T436614]]) === 2026-09-07 === * 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli === 2026-09-04 === * 16:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api * 16:21 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api === 2026-09-02 === * 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-6 * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-6 * 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-5 * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-5 * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-4 * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-4 === 2026-08-31 === * 18:03 taavi: shut down old elasticsearch v7 infrastructure [[phab:T401818|T401818]] === 2026-08-30 === * 20:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:26 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:43 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:17 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:15 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:03 dhinus: restart harbor in tools-harbor-2 with "docker compose restart" === 2026-08-26 === * 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 10:00 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:17 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-24 === * 10:46 taavi: disable access to elastic cluster as announced on cloud-announce [[phab:T401818|T401818]] === 2026-08-20 === * 16:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:41 dcaro: tools-puppetserver-01 was out of memory because of java and killed sssd backend * 15:33 dcaro: restart tools-puppetserver-01, unresponsive * 14:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:38 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:29 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:48 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 13:42 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:03 dcaro: truncate haproxy.log to 1G (out of space) on haproxy-8 === 2026-08-14 === * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-gateway-3 * 10:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-gateway-3 * 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8, tools-k8s-gateway-2 * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8, tools-k8s-gateway-2 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7, tools-k8s-gateway-1 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7, tools-k8s-gateway-1 === 2026-08-13 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 11:25 taavi: update striker to 2026-08-13-111038-production * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 09:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2026-08-12 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 13:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-05 === * 14:43 taavi: update striker to 2026-08-04-183239-production === 2026-08-04 === * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:22 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:15 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-03 === * 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, t * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 17:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers ([[phab:T433426|T433426]]) * 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T433426|T433426]]) * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-31 === * 13:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:03 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:36 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:28 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:18 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 15:10 godog: hard reboot tools-bastion-15 * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:05 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:04 dcaro: reboot tools-bastion-15, getting out of memory * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:00 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:29 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 09:24 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 09:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 09:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 09:16 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 * 09:16 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 === 2026-07-29 === * 09:54 godog: test dumps-nfs failover - [[phab:T432325|T432325]] * 09:48 godog: enable 'dumps_use_nfs_lb: true' for nfs worker prefix - [[phab:T432325|T432325]] * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 08:56 godog: enable 'dumps_use_nfs_lb: true' for bastion/static/mail prefixes - [[phab:T432325|T432325]] === 2026-07-28 === * 08:25 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, t * 06:56 godog: resume nfs workers roll restart - [[phab:T432325|T432325]] * 06:55 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs === 2026-07-27 === * 17:18 dcaro: increase object storage object number quota to 1M ([[phab:T433281|T433281]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:06 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 11:39 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 11:37 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) * 08:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) === 2026-07-23 === * 17:11 dcaro: resize haproxy-8 * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:55 dcaro: resize haproxy-7 * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32 cores, 65536 ram * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 32 cores, 65536 ram * 12:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-07-20 === * 12:28 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 12:22 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:43 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:33 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-16 === * 15:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-5 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-5 * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:38 taavi: failing over redis to new trixie cluster [[phab:T360704|T360704]] * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-6 * 11:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-6 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-7 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-7 * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-cumin-1 * 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-cumin-1 * 01:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 19:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 19:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 18:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-cumin' * 18:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-cumin' * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:26 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-8.tools.eqiad1.wikimedia.cloud * 13:49 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-8.tools.eqiad1.wikimedia.cloud * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:00 dcaro: restart haproxy on haproxy-8 removing tune.disable-zero-copy-forwarding from config === 2026-07-12 === * 21:01 dcaro: restart haproxy on haproxy-8 * 06:28 dcaro: restart haproxy on haproxy-8 === 2026-07-11 === * 17:34 dcaro: rebooting haproxy-8 === 2026-07-09 === * 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 12:53 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:27 taavi: rebooting bastions for kernel upgrades * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 10:18 taavi: deploy latest striker with [[phab:T430120|T430120]] [[phab:T409341|T409341]] fixes * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 17:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 08:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) === 2026-06-24 === * 16:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-15 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 12:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:45 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 11:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 09:53 godog: delete tools-checker puppet prefix - [[phab:T313030|T313030]] * 09:52 godog: delete tools-checker-5 - [[phab:T313030|T313030]] === 2026-06-11 === * 13:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-09 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 11:01 taavi: starting Kyverno upgrade [[phab:T427928|T427928]] === 2026-06-04 === * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:06 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:02 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry === 2026-06-02 === * 15:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 10:33 taavi: migrate acme-chief, puppetdb to nftables [[phab:T427799|T427799]] * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 20:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 20:33 andrewbogott: rebuilt pywikibot for v11.3.0 [[phab:T426547|T426547]] * 12:12 godog: begin roll-restart webservices with new default requests - [[phab:T420565|T420565]] * 12:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:02 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:11 godog: delete obsolete limitranges - [[phab:T427352|T427352]] === 2026-05-21 === * 12:46 taavi: renew prometheus certificate * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:48 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 00:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-05-20 === * 23:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 10:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-16 === * 00:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers === 2026-05-15 === * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 19:02 taavi: rebooting bastions and k8s workers to pick up kernel updates === 2026-05-14 === * 16:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-13 === * 12:07 godog: resume restarting webservices using default memory requests - [[phab:T420565|T420565]] * 08:46 godog: restart sample webservices with new memory requests https://phabricator.wikimedia.org/P92497 - [[phab:T420565|T420565]] * 08:36 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 22:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2026-05-11 === * 00:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 00:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-07 === * 12:02 taavi: draining tools-k8s-worker-106 to investigate [[phab:T425172|T425172]] === 2026-05-05 === * 04:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 02:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 10:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 15:59 andrewbogott: hard rebooting tools-puppetserver-01.tools, it seems to have crashed * 09:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:12 taavi: uninstall ingress-nginx-gen2 from the cluster [[phab:T392356|T392356]] * 08:08 taavi: delete all ingress objects [[phab:T392356|T392356]] === 2026-04-21 === * 14:06 taavi: save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json [[phab:T392356|T392356]] * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-16 === * 13:09 taavi: bump istio traffic percentage 75% -> 100% [[phab:T392356|T392356]] === 2026-04-15 === * 10:45 taavi: bump istio traffic percentage 50% -> 75% [[phab:T392356|T392356]] === 2026-04-13 === * 09:11 taavi: bump istio traffic percentage 25% -> 50% [[phab:T392356|T392356]] * 07:33 taavi: bump istio traffic percentage 10% -> 25% [[phab:T392356|T392356]] === 2026-04-10 === * 14:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-04-07 === * 23:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 15:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrewbogott: replacing etcd nodes with bookworm-based VMs * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 18:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-30 === * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 dcaro: removing wal from prometheus nodes to restart them === 2026-03-26 === * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-25 === * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04 * 13:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04 === 2026-03-24 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 11:16 taavi: send 10% of traffic to istio [[phab:T392356|T392356]] * 10:53 taavi: send 5% of traffic to istio [[phab:T392356|T392356]] * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 20:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:07 taavi: fixing 2 tools still running ruby2.1 image to use that instead of 'ruby2' in service.manifest * 08:52 taavi: fixing 2 tools still running ruby2.5 image to use that instead of 'ruby25' in service.manifest * 08:49 taavi: fixing 12 tools still running node6 image to use that instead of 'nodejs' in service.manifest * 08:38 taavi: fixing 12 tools still running golang1.11 image to use that instead of 'golang111' in service.manifest * 08:36 taavi: fixing 60 tools still running python3.4 image to use 'python3.4' instead of 'python' in service.manifest === 2026-03-18 === * 12:00 taavi: restarting existing web services to backfill HTTPRoute resources [[phab:T392356|T392356]] * 07:37 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:37 filippo@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-83.tools.eqiad1.wikimedia.cloud to the cluster * 07:23 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T419824|T419824]]) === 2026-03-17 === * 12:43 taavi: shutdown tools-package-builder-04 [[phab:T401819|T401819]] === 2026-03-15 === * 03:10 andrewbogott: rebooting tools-redis-6, VM is in state ERROR === 2026-03-13 === * 22:04 taavi: reboot tools-bastion-15 [[phab:T420044|T420044]] * 19:06 taavi: reboot tools-bastion-14 [[phab:T420044|T420044]] === 2026-03-12 === * 13:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 13:50 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2026-03-10 === * 11:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:30 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster === 2026-03-06 === * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-03-05 === * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-03-04 === * 20:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:44 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:29 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:14 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:13 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:12 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=97) * 19:11 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:08 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:17 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:38 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 14:54 dcaro: increase object quota to 400k ([[phab:T418528|T418528]]) * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2026-03-03 === * 20:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 20:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-25 === * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:09 taavi: taavi@tools-imagebuilder-2:~$ sudo docker system prune -a # reclaiming disk space * 14:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 === 2026-02-24 === * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-20 === * 20:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 19:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} === 2026-02-11 === * 21:49 taavi: remove hiera override still allowing ssh agent forwarding onto toolforge bastions [[phab:T198138|T198138]] === 2026-02-05 === * 19:08 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:48 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:42 volans: re-enabling puppet on NFS workers to update the infra-tracing-nfs === 2026-02-04 === * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 * 14:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 === 2026-02-03 === * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 === 2026-01-28 === * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-14 === * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 15:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) * 13:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) === 2026-01-12 === * 17:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 15:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 14:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 13:48 andrewbogott: removing tools-k8s-etcd-24 in prep for rebuilding cloudvirtlocal1003 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:01 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 22:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 200k * 11:05 godog: bump object quota to 500G === 2025-12-17 === * 17:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.6.3 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) === 2025-12-15 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.17.0 ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T412695|T412695]]) === 2025-12-14 === * 02:14 andrewbogott: running 'kubectl rollout restart -n envvars-admission deployment/envvars-admission' in response to an envvars alert === 2025-12-11 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) === 2025-12-03 === * 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) === 2025-12-02 === * 20:22 andrewbogott: stop/starting harbordb1 to fix presumed mtu mismatch * 20:06 andrewbogott: rebooting tools-harbordb1 to aid with host draining * 08:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.11.3 ([[phab:T399313|T399313]]) === 2025-12-01 === * 22:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 16:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:26 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-27 === * 12:15 volans: [continue] on the haproxy nodes * 12:15 volans: temporarily disabling puppet to deploy gerrit {{Gerrit|1211610}} === 2025-11-26 === * 14:48 volans: enabled infra-tracing-nfs on all nfs workers after testing it on few hosts * 09:46 dhinus: restarting tools-db-6 to apply a config change [[phab:T409922|T409922]] === 2025-11-25 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 02:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:24 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 10:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 18:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 * 17:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:55 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:45 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:36 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:56 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:51 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:37 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:37 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 15:17 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:01 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 14:46 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:40 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-19 === * 16:19 andrewbogott: increased object count quota to 100,000 * 16:03 andrewbogott: increased object storage quota to 200GB * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-11-18 === * 18:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-harbor-2 (cluster eqiad1) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-harbor-2 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-9 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-9 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-113 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-113 (cluster eqiad1) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-112 (cluster eqiad1) * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-112 (cluster eqiad1) * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-81 * 14:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-81 * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-80 * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-80 * 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-8 (cluster eqiad1) * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-8 (cluster eqiad1) * 12:05 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:56 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:24 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:19 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.instance.stop_start (exit_code=97) vm tools-legaci-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legaci-redirector-3 (cluster eqiad1) === 2025-11-17 === * 18:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T409981|T409981]]) * 10:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T409981|T409981]]) === 2025-11-14 === * 16:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-4 ([[phab:T409287|T409287]]) * 16:26 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-4 ([[phab:T409287|T409287]]) === 2025-11-13 === * 15:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-bastion-14 (cluster eqiad1) * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-bastion-14 (cluster eqiad1) * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-8 (cluster eqiad1) === 2025-11-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-11 === * 15:28 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 ([[phab:T399313|T399313]]) === 2025-11-10 === * 22:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 22:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:16 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:14 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:08 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest:latest * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-07 === * 11:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:35 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 ([[phab:T409287|T409287]]) * 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 ([[phab:T409287|T409287]]) === 2025-11-06 === * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 taavi: taavi@tools-bastion-15:~ $ sudo loginctl terminate-user damian * 14:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:53 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) === 2025-11-04 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 17:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-10-30 === * 18:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:39 taavi: kick off script to rebuild all pre-built images, including [[phab:T407707|T407707]] * 16:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 16:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 16:27 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T408669|T408669]]) * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6 * 12:48 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6 * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:07 taavi: delete paws, paws-master security groups, long obsolete as paws is now in a separae project * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-10-27 === * 22:09 taavi: copy toolviews database hiera data to a place where haproxy nodes can see them [[phab:T408454|T408454]] * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:16 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-24 === * 20:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t * 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k * 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106 * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106 * 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103 * 13:37 andrewbogott: rebooting clouddumps100[12] for [[phab:T407110|T407110]] === 2025-10-23 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:11 taavi: deleting old nginx front proxy instances [[phab:T283948|T283948]] * 10:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 15:56 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3 * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:35 taavi: moving toolforge traffic to haproxy directly [[phab:T283948|T283948]] * 07:00 godog: delete tools-nfs-2 - [[phab:T404584|T404584]] === 2025-10-21 === * 08:53 godog: shut down tools-nfs-2 - [[phab:T404584|T404584]] * 07:52 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 07:49 godog: resize tools-nfs-3 to match tools-nfs-2 (g4.cores16.ram64.disk20.10xiops) - [[phab:T404584|T404584]] * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-20 === * 16:31 taavi: make logrotate run hourly on haproxy nodes [[phab:T284558|T284558]] === 2025-10-16 === * 12:01 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-15 === * 08:03 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 08:00 godog: resize tools-nfs-3 - [[phab:T404584|T404584]] === 2025-10-14 === * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:45 godog: update nfs-tools.wmcloud.org and nfs.svc.toolforge.org proxied to point to tools-nfs-3 === 2025-10-13 === * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:17 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, too * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:13 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:13 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:10 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:01 godog: switch NFS from tools-nfs-2 to tools-nfs-3 - [[phab:T404584|T404584]] * 07:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81 === 2025-10-10 === * 09:22 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 * 09:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 === 2025-10-09 === * 08:21 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 08:15 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2025-10-08 === * 21:47 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 21:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 12:27 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]] * 10:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 06:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 * 06:43 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 === 2025-10-07 === * 18:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11 * 15:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69 * 15:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69 * 14:51 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 14:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:58 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-06 === * 12:06 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7 * 11:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7 * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 08:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:18 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 * 07:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 === 2025-10-03 === * 12:51 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-6 * 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-6 * 12:45 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-5 * 09:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-5 === 2025-10-02 === * 13:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 09:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 * 08:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 === 2025-10-01 === * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-30 === * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 * 08:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 === 2025-09-29 === * 13:23 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 13:23 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 11:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 07:00 godog: kick stuck nfs workers from clouddumps1001 === 2025-09-28 === * 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:13 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:07 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:07 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-25 === * 18:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 20:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:30 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:07 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 13:49 dcaro: patched all tools with new resource defaults, everything looks good * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro: depolyed jobs-api change to default resources, patching existing jobs * 13:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 03:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 === 2025-09-23 === * 20:08 andrewbogott: creating puppetdbpostgres and adding it to tools-puppetdb-2 to store postgres data; the root volume of that VM was filling up and causing widespread puppet issues * 01:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-09-22 === * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-21 === * 09:17 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 03:16 dcaro: acking and silencing CPU capacity alerts to handle on Monday, they should not page * 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 01:46 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-113.tools.eqiad1.wikimedia.cloud to the cluster * 01:36 andrewbogott: adding additional worker node in response to repeated capacity alerts * 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-09-19 === * 13:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 13:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 === 2025-09-18 === * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:52 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 08:34 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 06:47 taavi: delete tools-sgebastion-10 [[phab:T314665|T314665]] === 2025-09-17 === * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 12:53 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 * 08:08 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 === 2025-09-16 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:57 taavi: delete tools-sgebastion puppet prefix [[phab:T314665|T314665]] * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 taavi: shutdown tools-sgebastion-10 [[phab:T314665|T314665]] * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-13 * 14:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-13 * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-12 * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-12 * 07:11 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 06:51 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-15 === * 16:22 taavi: reboot old bastions to kick long-living connections into newer ones * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:47 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 * 12:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 === 2025-09-12 === * 08:49 taavi: pointing login.toolforge.org to tools-bastion-15 [[phab:T392510|T392510]] * 08:33 taavi: pointing dev.toolforge.org to tools-bastion-14 [[phab:T392510|T392510]] * 07:14 godog: uncordon tools-k8s-worker-nfs-53 after failed cookbook (?) yesterday === 2025-09-11 === * 14:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:36 godog: drain/reboot tools-k8s-worker-nfs-46 - [[phab:T404322|T404322]] * 14:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:22 andrewbogott: actually I didn't drain tools-k8s-worker-nfs-53 because the alert cleared on its own * 14:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-53 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 14:21 andrewbogott: draining/rebooting tools-k8s-worker-nfs-53 because of procs in D state * 13:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 13:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-09-10 === * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:45 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) === 2025-09-09 === * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:55 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) ([[phab:T404047|T404047]]) * 08:55 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot ([[phab:T404047|T404047]]) === 2025-09-08 === * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:16 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:40 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests === 2025-09-06 === * 23:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 * 23:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 === 2025-09-05 === * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 19:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:52 dcaro: added 'disable-ssl' to tools replica.my.cnf === 2025-09-03 === * 17:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 17:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-29 === * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting === 2025-08-28 === * 16:52 taavi: rebuild tcl, mariadb images on top of trixie [[phab:T400256|T400256]] * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:05 taavi: copy missing aptly packages to trixie-<nowiki>{</nowiki>tools,toolsbeta<nowiki>}</nowiki> * 11:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-26 === * 13:42 dcaro: extended object storage quota to 100G ([[phab:T402923|T402923]]) * 10:25 dhinus: shut down tools-harbor-1 (no longer used) === 2025-08-25 === * 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-81 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-81 === 2025-08-21 === * 12:28 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:31 godog: reboot nfs workers to reset processes stuck in D state * 07:28 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 04:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 * 03:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 === 2025-08-20 === * 08:09 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) === 2025-08-19 === * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:37 dcaro: flipped the tools-harbor.wmcloud.org endpoint to point to tools-harbor-2 ([[phab:T350687|T350687]]) * 14:22 Raymond_Ndibe: setting tools-harbor-1 as read-only ([[phab:T350687|T350687]]) * 13:24 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 13:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.16.0 ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/metrics-server:v0.7.2 ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud === 2025-08-16 === * 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-111 * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-111 * 21:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-111 * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-111 === 2025-08-15 === * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 19:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 * 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 === 2025-08-14 === * 15:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 11:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 02:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:36 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) * 11:18 taavi: delete tools-prometheus-6, shutdown for a while * 08:51 godog: bounce stashbot * 08:33 godog: refresh machine-id on tools-k8s-worker-[102-103,105-112].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-3,5,7-14,16-17,19,21-24,26-27,32-48,50,53-55 ,57-58,61,65-82].tools.eqiad1.wikimedia.cloud === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:34 taavi: building initial trixie based images [[phab:T400255|T400255]] * 12:50 dcaro: redepoly kyverno ([[phab:T394787|T394787]]) * 12:49 dcaro: manually migrate cleanuppolicies.kyverno.io and clustercleanuppolicies.kyverno.io (using kyverno cli) ([[phab:T394787|T394787]]) * 10:01 dcaro: starting upgrade for kyverno ([[phab:T394787|T394787]]) * 10:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 === 2025-08-08 === * 06:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 * 14:20 andrewbogott: draining and rebooting tools-k8s-worker-nfs-67 * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:54 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:41 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:39 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:38 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-05 === * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 03:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 02:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 * 11:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 === 2025-08-01 === * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 04:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-07-30 === * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 15:56 andrewbogott: draining and restarting tools-k8s-worker-nfs-74 * 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrewbogott: draining and restarting tools-k8s-worker-nfs-58 and tools-k8s-worker-nfs-32 * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:16 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-82.tools.eqiad1.wikimedia.cloud to the cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-81.tools.eqiad1.wikimedia.cloud to the cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.quota_increase * 12:39 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-80.tools.eqiad1.wikimedia.cloud to the cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:00 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97) * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:28 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:07 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:59 wmbot~dcaro@acme: Added a new k8s worker tools-k8s-worker-112.tools.eqiad1.wikimedia.cloud to the cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:14 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-07-28 === * 20:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:58 taavi: update pywikibot image to 10.2.0 [[phab:T396933|T396933]] === 2025-07-26 === * 07:16 wmbot~root@toolforge: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:16 wmbot~root@toolforge: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-23 === * 18:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-21 === * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-07-19 === * 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 13:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-07-18 === * 10:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 === 2025-07-14 === * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78 === 2025-07-13 === * 03:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 * 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 === 2025-07-11 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 * 09:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 === 2025-07-09 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:55 dcaro: adding arch arm64 to all toolforge repos ([[phab:T398016|T398016]]) * 09:40 dcaro: added arch arm64 to jessie-tools repo ([[phab:T398016|T398016]]) === 2025-07-08 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-07 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 17:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 16:39 dcaro: pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest * 16:05 dcaro: clearing images from tools-imagebuilder-2 as it's out of space * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-06 === * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 * 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 === 2025-07-05 === * 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs === 2025-07-04 === * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-07-03 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-02 === * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55 * 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-06-30 === * 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) === 2025-06-28 === * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:08 dcaro: left a tmux running with a script to restart nginx if stuck * 09:59 dcaro: restarted nginx in tools-static === 2025-06-27 === * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 === 2025-06-26 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 * 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 === 2025-06-24 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-23 === * 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]] === 2025-06-22 === * 00:09 andrewbogott: rebooting tools-prometheus-8 === 2025-06-21 === * 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of D state * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-19 === * 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools === 2025-06-18 === * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable === 2025-06-16 === * 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 === 2025-06-14 === * 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-06-12 === * 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh) * 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-11 === * 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 === 2025-06-10 === * 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved === 2025-06-06 === * 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark * 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-06-05 === * 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert * 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-30 === * 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 07:38 taavi: reboot tools-static-15 to unstuck NFS things === 2025-05-24 === * 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 * 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 === 2025-05-23 === * 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 * 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 === 2025-05-22 === * 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:15 dcaro: restart tools-static nginx due to nfs hiccup * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7 * 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 * 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]] === 2025-05-21 === * 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh) * 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead === 2025-05-20 === * 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97) * 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown * 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-19 === * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-14 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36 * 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-05-12 === * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512 ([[phab:T393686|T393686]]) * 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 * 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 === 2025-05-10 === * 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo<nowiki>{</nowiki>,.socket<nowiki>}</nowiki> # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]]) * 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-<nowiki>{</nowiki>pam,sudo<nowiki>}</nowiki>.service && systemctl restart sssd-pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket # try to reset the rate limits this way ([[phab:T393732|T393732]]) * 16:22 lucaswerkmeister: systemctl restart sssd-<nowiki>{</nowiki>pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>,sudo<nowiki>}</nowiki>.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient) * 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam<nowiki>{</nowiki>,<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket<nowiki>}</nowiki> # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]? === 2025-05-09 === * 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake * 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory) * 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs === 2025-05-08 === * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state * 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-07 === * 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2 * 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2 * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:33 dcaro: released jobs-cli 16.1.12 * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 === 2025-05-04 === * 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06) === 2025-05-02 === * 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]] * 13:55 taavi: reboot tools-static-15 === 2025-04-28 === * 13:07 dhinus: tools-db-4: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:06 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:05 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T318479|T318479]] === 2025-04-24 === * 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12 * 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing "sudo: PAM account management error: Authentication service cannot retrieve authentication info" * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:51 taavi: add missing ICMPv6 security group rule to 'default' group * 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]] === 2025-04-23 === * 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles === 2025-04-21 === * 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]] === 2025-04-17 === * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-11 === * 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-10 === * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2025-04-09 === * 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff) * 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-04-07 === * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-06 === * 02:12 andrewbogott: truncating large logfiles on tools nfs === 2025-04-04 === * 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes === 2025-04-03 === * 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 === 2025-04-02 === * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 === 2025-04-01 === * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 === 2025-03-31 === * 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 * 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 === 2025-03-28 === * 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]]) * 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]]) * 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]]) * 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]]) === 2025-03-27 === * 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster * 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 === 2025-03-25 === * 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 * 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2 * 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 * 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-03-22 === * 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-18 === * 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) * 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) === 2025-03-17 === * 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) === 2025-03-16 === * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 === 2025-03-15 === * 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]] === 2025-03-13 === * 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 === 2025-03-12 === * 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o * 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-03-07 === * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-03-06 === * 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-03-05 === * 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values) * 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes * 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-03-04 === * 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual) * 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-03 === * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-01 === * 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 === 2025-02-27 === * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-02-26 === * 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-25 === * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 === 2025-02-24 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-21 === * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-02-20 === * 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-19 === * 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-02-18 === * 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]]) * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]]) === 2025-02-17 === * 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-10 === * 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-02-09 === * 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes === 2025-02-07 === * 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]] * 17:58 andrewbogott: "SET GLOBAL read_only=OFF; " on tools-db-4; both -5 and -4 were set to read only. No idea why or how... * 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 === 2025-02-06 === * 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage * 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-03 === * 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 === 2025-02-01 === * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102 * 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]] * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102 * 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]] * 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 * 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 === 2025-01-31 === * 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]] === 2025-01-29 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-27 === * 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-26 === * 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster * 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster * 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 * 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 === 2025-01-24 === * 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 === 2025-01-23 === * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs === 2025-01-22 === * 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2025-01-18 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2025-01-17 === * 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing) === 2025-01-15 === * 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-13 === * 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:49 dcaro: restart prometheus to pick up the new ips for vms and such * 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 * 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 * 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots === 2025-01-12 === * 09:53 taavi: hard reboot tools-k8s-worker-nfs-55 === 2025-01-08 === * 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22 * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22 * 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70 * 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70 * 14:16 dcaro: reboot tools-static-15 nfs is stuck === 2025-01-07 === * 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor === 2025-01-06 === * 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-01-05 === * 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13 === 2025-01-03 === * 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 * 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69 * 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69 === 2025-01-02 === * 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 === 2025-01-01 === * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 === 2024-12-13 === * 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 * 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 === 2024-12-12 === * 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2024-12-06 === * 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]]) * 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]]) * 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing * 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main === 2024-11-29 === * 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-27 === * 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns # update resolv.conf in coredns containers === 2024-11-26 === * 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:22 dcaro: rebooting k8s-control-9 * 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:17 dcaro: rebooting k8s-control-8 * 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:14 dcaro: restarting tools-k8s-worker-nfs-72 * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 09:12 dcaro: restarting tools-k8s-worker-nfs-70 * 09:11 dcaro: restarting tools-k8s-worker-nfs-50 * 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 * 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]]) * 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]]) === 2024-11-25 === * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2024-11-23 === * 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]]) * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]]) * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-18 === * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-15 === * 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]]) * 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) === 2024-11-14 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-10 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]]) === 2024-11-06 === * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 dcaro: added api.svc.toolforge.org dns record entry * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-22 === * 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 * 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]] === 2024-10-16 === * 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-14 === * 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]]) * 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 * 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-10-09 === * 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup * 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating * 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]] * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-04 === * 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-02 === * 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-01 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:28 dcaro: updated ci image with latest precommit versions * 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-30 === * 18:25 taavi: run striker migrations [[phab:T359428|T359428]] === 2024-09-28 === * 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2024-09-24 === * 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-20 === * 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]]) * 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network * 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-17 === * 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64 * 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63 * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62 * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60 * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6 * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56 * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49 * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31 * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30 * 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29 * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28 * 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster * 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]]) === 2024-09-16 === * 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-09-13 === * 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) === 2024-09-12 === * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) * 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) === 2024-09-11 === * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-09 === * 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager === 2024-09-06 === * 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6 * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-09-05 === * 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-03 === * 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 * 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 === 2024-09-02 === * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15 * 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15 * 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2 * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster * 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-08-26 === * 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster * 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster * 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 11:06 dcaro: manually deleted the coredns pods that had been around for 4d * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:18 dcaro: scale up cordens deployment to 4 replicas === 2024-08-21 === * 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-19 === * 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 === 2024-08-15 === * 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 * 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 === 2024-08-13 === * 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-08-12 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-06 === * 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2024-08-05 === * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot * 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected. * 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55 * 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc * 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]] * 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 * 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 === 2024-07-30 === * 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2024-07-29 === * 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) * 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance * 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission * 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 17:00 dcaro: moving the toolforge apt repo to tools-services-06 * 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud * 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-19 === * 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6 * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-07-18 === * 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16 * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16 * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-11 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1 * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0 * 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]] * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]]) * 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]]) * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23 * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23 * 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1 * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24 * 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24 * 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22 * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22 * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]]) * 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]] * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 === 2024-06-25 === * 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-> dict[str, Any]` type annotations causing [[phab:T368463|T368463]] * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104 * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103 * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56 * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55 * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54 * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53 * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52 * 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51 * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53 * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52 * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7 * 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7 * 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8' * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8' * 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9 * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9 * 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47 * 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44 * 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44 * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42 * 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41 * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39 * 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38 * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37 * 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36 * 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35 * 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35 * 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34 * 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33 * 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35 * 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33 * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33 * 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33 === 2024-06-24 === * 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots. * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32 * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29 * 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29 * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28 * 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28 * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27 * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27 * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10 * 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10 * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13 * 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13 * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26 * 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22 * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22 * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21 * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21 * 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19 * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20 * 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17 * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14 * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13 * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13 * 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12 * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12 * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7 * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8 * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8 * 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8 * 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15 * 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4 * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4 * 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10 * 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10 * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9 * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7 * 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7 * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3 * 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5 * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7 * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7 * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7 * 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43 * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7 * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7 * 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7 * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43 * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7 * 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6 * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6 * 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1 * 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5 * 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5 * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1 * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6 * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2 * 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2 * 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2 * 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2 * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2 * 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05 * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8 * 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8 * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5 * 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5 * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-06-20 === * 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]] * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-14 === * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]]. * 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 13:45 taavi: hard reboot tools-k8s-control-7 * 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:57 dcaro: cleaning old maintain-kubeusers configmaps * 10:45 dcaro: cleaning up old resourcequotas === 2024-06-10 === * 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno === 2024-06-07 === * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 dcaro: deploying toolforge-webservice 0.103.6 * 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13 * 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12 === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-05-29 === * 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api * 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-28 === * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-25 === * 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-22 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-15 === * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO * 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9 * 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 === 2024-05-07 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-05 === * 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]] === 2024-04-24 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-17 === * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13 * 10:48 dcaro: rebooting tools-k8s-worker-nfs-1 === 2024-04-16 === * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0' * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0' === 2024-04-15 === * 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-11 === * 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-09 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission * 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6 === 2024-04-08 === * 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 * 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5 === 2024-04-05 === * 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]] === 2024-04-02 === * 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]]) * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05 === 2024-03-28 === * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05 * 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]] * 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06 * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06 * 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' === 2024-03-27 === * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01 === 2024-03-26 === * 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion' * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion' * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11 * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11 * 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]] === 2024-03-25 === * 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector * 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud === 2024-03-22 === * 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error) === 2024-03-21 === * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4 * 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3 * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04 * 12:30 taavi: move checker service address to tools-checker-5 * 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker' * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' === 2024-03-19 === * 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images) * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14 * 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]] * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]]) === 2024-03-18 === * 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 taavi: restart harbor services after docker service restart * 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1 * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1 * 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]] * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]] === 2024-03-15 === * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48' * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48' * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01 * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]] * 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]] === 2024-03-13 === * 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable. === 2024-03-12 === * 12:38 taavi: hard reboot tools-prometheus-6 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-11 === * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]] === 2024-03-09 === * 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs === 2024-03-08 === * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-07 === * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-06 === * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates * 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs === 2024-03-05 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]]) * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud === 2024-03-04 === * 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state === 2024-03-03 === * 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state) === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-29 === * 14:36 dcaro: deploy webservice 0.103.3 === 2024-02-28 === * 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]]) * 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) === 2024-02-23 === * 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2 * 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]] === 2024-02-22 === * 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster * 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 === 2024-02-21 === * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4 * 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster === 2024-02-20 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster * 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101 * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102 * 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102 * 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster * 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100 * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster * 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96 * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95 * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95 * 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92 * 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster * 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91 * 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91 * 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90 * 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88 === 2024-02-19 === * 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster * 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84 * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83 * 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81 * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81 * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-16 === * 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80 * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77 * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77 === 2024-02-15 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4 * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4 * 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76 * 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75 * 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster === 2024-02-14 === * 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30 * 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3 * 16:30 taavi: reboot tools-sgeexec-10-23 due to high load * 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74 * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster * 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71 * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71 * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster * 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70 * 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70 * 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69 * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68 * 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68 === 2024-02-13 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster * 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65 * 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64 * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64 === 2024-02-12 === * 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster * 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster * 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61 * 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60 * 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60 * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster * 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58 * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58 * 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54 * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15 * 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-11 === * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-02-09 === * 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule * 13:06 taavi: reboot tools-sgecron-2 due to high load * 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config * 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 * 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes === 2024-02-08 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 * 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42 * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42 === 2024-02-07 === * 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers * 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers === 2024-02-06 === * 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-01-31 === * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-30 === * 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster * 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster * 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster * 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41 * 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41 * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40 * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40 * 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39 * 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39 * 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38 * 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38 * 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37 * 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37 * 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]]) * 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]]) * 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]] * 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]]) * 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]]) * 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]]) === 2024-01-29 === * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud * 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud * 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster * 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33 * 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33 * 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32 * 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32 * 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31 * 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30 * 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster * 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB * 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB * 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-26 === * 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta === 2024-01-25 === * 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1 === 2024-01-23 === * 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22 * 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-18 === * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17 === 2024-01-17 === * 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]] * 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]]) * 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]]) * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]] === 2024-01-15 === * 09:18 taavi: reboot stuck tools-k8s-worker-84 === 2024-01-12 === * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12' * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12' === 2024-01-11 === * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-10 === * 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:17 taavi: reboot tools-k8s-worker-98 === 2024-01-09 === * 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor -- [[phab:T354714|T354714]] * 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it) * 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder * 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder * 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:13 taavi: reboot tools-sgeexec-10-17 due to high load === 2024-01-08 === * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28 * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 taavi: reboot tools-sgeexec-10-21 === 2024-01-05 === * 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-04 === * 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3 === 2024-01-03 === * 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs * 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage === 2024-01-02 === * 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]]) * 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]]) * 10:37 dcaro: hard reboot tools-harbor-1 * 10:13 dhinus: hard reboot tools-harbor-1 === 2024-01-01 === * 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]] ==Archives== * [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014) * [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017) * [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019) * [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021) * [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023) </noinclude> {{SAL|Project Name=tools}} <noinclude>[[Category:SAL]]</noinclude> 0mwo5asz9epwj2wmke49iym32osx01m 2466286 2466285 2026-10-11T07:40:17Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2466286 wikitext text/x-wiki === 2026-10-11 === * 07:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 07:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-09 === * 18:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.3.1 ([[phab:T414595|T414595]]) * 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.1.3 ([[phab:T414595|T414595]]) * 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/kube-state-metrics:v8.1.3 ([[phab:T414595|T414595]]) === 2026-10-08 === * 17:01 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:58 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-07 === * 15:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T440326|T440326]]) * 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T440326|T440326]]) * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-06 === * 17:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2026-10-05 === * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T439237|T439237]]) * 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T439237|T439237]]) * 15:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T439237|T439237]]) * 14:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T439237|T439237]]) * 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 11:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-10-02 === * 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 13:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) === 2026-10-01 === * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 14:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-30 === * 14:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 09:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-29 === * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-25 === * 18:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:31 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-90.tools.eqiad1.wikimedia.cloud to the cluster * 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-87 * 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-87 * 13:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 13:45 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster * 13:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) * 13:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-115 ([[phab:T438255|T438255]]) * 13:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-115 ([[phab:T438255|T438255]]) * 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:57 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-89.tools.eqiad1.wikimedia.cloud to the cluster * 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 01:43 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster * 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) === 2026-09-24 === * 20:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 20:26 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster * 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) * 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 20:17 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-114.tools.eqiad1.wikimedia.cloud to the cluster * 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T438255|T438255]]) * 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 20:05 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-89.tools.eqiad1.wikimedia.cloud to the cluster * 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:20 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-88.tools.eqiad1.wikimedia.cloud to the cluster * 19:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 18:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:38 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-87.tools.eqiad1.wikimedia.cloud to the cluster * 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:02 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-86.tools.eqiad1.wikimedia.cloud to the cluster * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 192 cores, 262144 ram ([[phab:T439157|T439157]]) * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 192 cores, 262144 ram ([[phab:T439157|T439157]]) * 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:38 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-85.tools.eqiad1.wikimedia.cloud to the cluster * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:22 andrew@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-84.tools.eqiad1.wikimedia.cloud to the cluster * 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T438255|T438255]]) * 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-23 === * 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:11 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:59 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:57 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:56 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-22 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-backup-1.tools.eqiad1.wikimedia.cloud * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-backup-1.tools.eqiad1.wikimedia.cloud * 10:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools--k8s-backup-1.tools.eqiad1.wikimedia.cloud * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools--k8s-backup-1.tools.eqiad1.wikimedia.cloud * 07:51 godog: correction that's tools-k8s-worker-nfs-74 * 07:48 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 07:45 godog: hard reboot tools-k8s-worker-nfs-70 - ran out of memory * 07:36 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-21 === * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 10:15 godog: reboot tools-k8s-worker-nfs-70 * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:39 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:34 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:33 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:33 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:33 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:33 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 08:30 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:29 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:23 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:14 dcaro: rebooting bastion, connections will close/fail for a few seconds * 07:41 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 07:36 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2026-09-18 === * 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 14:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 13:39 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) === 2026-09-16 === * 13:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-16 * 12:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 * 12:33 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 12:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2026-09-15 === * 18:09 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-09-14 === * 22:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 22:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-11 === * 09:08 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:58 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-10 === * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 14:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-08 === * 14:40 komla@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T436614|T436614]]) * 14:19 komla@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T436614|T436614]]) * 14:13 komla@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T436614|T436614]]) * 14:13 komla@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T436614|T436614]]) === 2026-09-07 === * 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli === 2026-09-04 === * 16:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api * 16:21 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api === 2026-09-02 === * 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-6 * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-6 * 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-5 * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-5 * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-4 * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-4 === 2026-08-31 === * 18:03 taavi: shut down old elasticsearch v7 infrastructure [[phab:T401818|T401818]] === 2026-08-30 === * 20:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:26 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:43 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:17 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:15 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:03 dhinus: restart harbor in tools-harbor-2 with "docker compose restart" === 2026-08-26 === * 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 10:00 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:17 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-24 === * 10:46 taavi: disable access to elastic cluster as announced on cloud-announce [[phab:T401818|T401818]] === 2026-08-20 === * 16:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:41 dcaro: tools-puppetserver-01 was out of memory because of java and killed sssd backend * 15:33 dcaro: restart tools-puppetserver-01, unresponsive * 14:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:38 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:29 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:48 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 13:42 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:03 dcaro: truncate haproxy.log to 1G (out of space) on haproxy-8 === 2026-08-14 === * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-gateway-3 * 10:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-gateway-3 * 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8, tools-k8s-gateway-2 * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8, tools-k8s-gateway-2 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7, tools-k8s-gateway-1 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7, tools-k8s-gateway-1 === 2026-08-13 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 11:25 taavi: update striker to 2026-08-13-111038-production * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 09:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2026-08-12 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 13:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-05 === * 14:43 taavi: update striker to 2026-08-04-183239-production === 2026-08-04 === * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:22 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:15 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-03 === * 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, t * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 17:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers ([[phab:T433426|T433426]]) * 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T433426|T433426]]) * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-31 === * 13:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:03 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:36 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:28 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:18 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 15:10 godog: hard reboot tools-bastion-15 * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:05 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:04 dcaro: reboot tools-bastion-15, getting out of memory * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:00 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:29 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 09:24 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 09:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 09:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 09:16 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 * 09:16 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 === 2026-07-29 === * 09:54 godog: test dumps-nfs failover - [[phab:T432325|T432325]] * 09:48 godog: enable 'dumps_use_nfs_lb: true' for nfs worker prefix - [[phab:T432325|T432325]] * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 08:56 godog: enable 'dumps_use_nfs_lb: true' for bastion/static/mail prefixes - [[phab:T432325|T432325]] === 2026-07-28 === * 08:25 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, t * 06:56 godog: resume nfs workers roll restart - [[phab:T432325|T432325]] * 06:55 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs === 2026-07-27 === * 17:18 dcaro: increase object storage object number quota to 1M ([[phab:T433281|T433281]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:06 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 11:39 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 11:37 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) * 08:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) === 2026-07-23 === * 17:11 dcaro: resize haproxy-8 * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:55 dcaro: resize haproxy-7 * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32 cores, 65536 ram * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 32 cores, 65536 ram * 12:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-07-20 === * 12:28 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 12:22 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:43 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:33 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-16 === * 15:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-5 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-5 * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:38 taavi: failing over redis to new trixie cluster [[phab:T360704|T360704]] * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-6 * 11:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-6 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-7 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-7 * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-cumin-1 * 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-cumin-1 * 01:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 19:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 19:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 18:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-cumin' * 18:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-cumin' * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:26 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-8.tools.eqiad1.wikimedia.cloud * 13:49 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-8.tools.eqiad1.wikimedia.cloud * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:00 dcaro: restart haproxy on haproxy-8 removing tune.disable-zero-copy-forwarding from config === 2026-07-12 === * 21:01 dcaro: restart haproxy on haproxy-8 * 06:28 dcaro: restart haproxy on haproxy-8 === 2026-07-11 === * 17:34 dcaro: rebooting haproxy-8 === 2026-07-09 === * 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 12:53 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:27 taavi: rebooting bastions for kernel upgrades * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 10:18 taavi: deploy latest striker with [[phab:T430120|T430120]] [[phab:T409341|T409341]] fixes * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 17:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 08:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) === 2026-06-24 === * 16:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-15 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 12:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:45 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 11:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 09:53 godog: delete tools-checker puppet prefix - [[phab:T313030|T313030]] * 09:52 godog: delete tools-checker-5 - [[phab:T313030|T313030]] === 2026-06-11 === * 13:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-09 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 11:01 taavi: starting Kyverno upgrade [[phab:T427928|T427928]] === 2026-06-04 === * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:06 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:02 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry === 2026-06-02 === * 15:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 10:33 taavi: migrate acme-chief, puppetdb to nftables [[phab:T427799|T427799]] * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 20:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 20:33 andrewbogott: rebuilt pywikibot for v11.3.0 [[phab:T426547|T426547]] * 12:12 godog: begin roll-restart webservices with new default requests - [[phab:T420565|T420565]] * 12:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:02 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:11 godog: delete obsolete limitranges - [[phab:T427352|T427352]] === 2026-05-21 === * 12:46 taavi: renew prometheus certificate * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:48 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 00:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-05-20 === * 23:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 10:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-16 === * 00:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers === 2026-05-15 === * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 19:02 taavi: rebooting bastions and k8s workers to pick up kernel updates === 2026-05-14 === * 16:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-13 === * 12:07 godog: resume restarting webservices using default memory requests - [[phab:T420565|T420565]] * 08:46 godog: restart sample webservices with new memory requests https://phabricator.wikimedia.org/P92497 - [[phab:T420565|T420565]] * 08:36 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 22:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2026-05-11 === * 00:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 00:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-07 === * 12:02 taavi: draining tools-k8s-worker-106 to investigate [[phab:T425172|T425172]] === 2026-05-05 === * 04:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 02:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 10:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 15:59 andrewbogott: hard rebooting tools-puppetserver-01.tools, it seems to have crashed * 09:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:12 taavi: uninstall ingress-nginx-gen2 from the cluster [[phab:T392356|T392356]] * 08:08 taavi: delete all ingress objects [[phab:T392356|T392356]] === 2026-04-21 === * 14:06 taavi: save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json [[phab:T392356|T392356]] * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-16 === * 13:09 taavi: bump istio traffic percentage 75% -> 100% [[phab:T392356|T392356]] === 2026-04-15 === * 10:45 taavi: bump istio traffic percentage 50% -> 75% [[phab:T392356|T392356]] === 2026-04-13 === * 09:11 taavi: bump istio traffic percentage 25% -> 50% [[phab:T392356|T392356]] * 07:33 taavi: bump istio traffic percentage 10% -> 25% [[phab:T392356|T392356]] === 2026-04-10 === * 14:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-04-07 === * 23:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 15:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrewbogott: replacing etcd nodes with bookworm-based VMs * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 18:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-30 === * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 dcaro: removing wal from prometheus nodes to restart them === 2026-03-26 === * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-25 === * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04 * 13:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04 === 2026-03-24 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 11:16 taavi: send 10% of traffic to istio [[phab:T392356|T392356]] * 10:53 taavi: send 5% of traffic to istio [[phab:T392356|T392356]] * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 20:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:07 taavi: fixing 2 tools still running ruby2.1 image to use that instead of 'ruby2' in service.manifest * 08:52 taavi: fixing 2 tools still running ruby2.5 image to use that instead of 'ruby25' in service.manifest * 08:49 taavi: fixing 12 tools still running node6 image to use that instead of 'nodejs' in service.manifest * 08:38 taavi: fixing 12 tools still running golang1.11 image to use that instead of 'golang111' in service.manifest * 08:36 taavi: fixing 60 tools still running python3.4 image to use 'python3.4' instead of 'python' in service.manifest === 2026-03-18 === * 12:00 taavi: restarting existing web services to backfill HTTPRoute resources [[phab:T392356|T392356]] * 07:37 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:37 filippo@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-83.tools.eqiad1.wikimedia.cloud to the cluster * 07:23 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T419824|T419824]]) === 2026-03-17 === * 12:43 taavi: shutdown tools-package-builder-04 [[phab:T401819|T401819]] === 2026-03-15 === * 03:10 andrewbogott: rebooting tools-redis-6, VM is in state ERROR === 2026-03-13 === * 22:04 taavi: reboot tools-bastion-15 [[phab:T420044|T420044]] * 19:06 taavi: reboot tools-bastion-14 [[phab:T420044|T420044]] === 2026-03-12 === * 13:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 13:50 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2026-03-10 === * 11:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:30 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster === 2026-03-06 === * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-03-05 === * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-03-04 === * 20:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:44 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:29 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:14 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:13 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:12 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=97) * 19:11 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:08 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:17 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:38 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 14:54 dcaro: increase object quota to 400k ([[phab:T418528|T418528]]) * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2026-03-03 === * 20:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 20:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-25 === * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:09 taavi: taavi@tools-imagebuilder-2:~$ sudo docker system prune -a # reclaiming disk space * 14:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 === 2026-02-24 === * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-20 === * 20:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 19:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} === 2026-02-11 === * 21:49 taavi: remove hiera override still allowing ssh agent forwarding onto toolforge bastions [[phab:T198138|T198138]] === 2026-02-05 === * 19:08 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:48 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:42 volans: re-enabling puppet on NFS workers to update the infra-tracing-nfs === 2026-02-04 === * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 * 14:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 === 2026-02-03 === * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 === 2026-01-28 === * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-14 === * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 15:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) * 13:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) === 2026-01-12 === * 17:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 15:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 14:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 13:48 andrewbogott: removing tools-k8s-etcd-24 in prep for rebuilding cloudvirtlocal1003 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:01 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 22:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 200k * 11:05 godog: bump object quota to 500G === 2025-12-17 === * 17:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.6.3 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) === 2025-12-15 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.17.0 ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T412695|T412695]]) === 2025-12-14 === * 02:14 andrewbogott: running 'kubectl rollout restart -n envvars-admission deployment/envvars-admission' in response to an envvars alert === 2025-12-11 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) === 2025-12-03 === * 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) === 2025-12-02 === * 20:22 andrewbogott: stop/starting harbordb1 to fix presumed mtu mismatch * 20:06 andrewbogott: rebooting tools-harbordb1 to aid with host draining * 08:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.11.3 ([[phab:T399313|T399313]]) === 2025-12-01 === * 22:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 16:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:26 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-27 === * 12:15 volans: [continue] on the haproxy nodes * 12:15 volans: temporarily disabling puppet to deploy gerrit {{Gerrit|1211610}} === 2025-11-26 === * 14:48 volans: enabled infra-tracing-nfs on all nfs workers after testing it on few hosts * 09:46 dhinus: restarting tools-db-6 to apply a config change [[phab:T409922|T409922]] === 2025-11-25 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 02:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:24 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 10:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 18:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 * 17:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:55 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:45 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:36 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:56 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:51 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:37 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:37 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 15:17 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:01 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 14:46 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:40 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-19 === * 16:19 andrewbogott: increased object count quota to 100,000 * 16:03 andrewbogott: increased object storage quota to 200GB * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-11-18 === * 18:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-harbor-2 (cluster eqiad1) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-harbor-2 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-9 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-9 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-113 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-113 (cluster eqiad1) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-112 (cluster eqiad1) * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-112 (cluster eqiad1) * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-81 * 14:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-81 * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-80 * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-80 * 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-8 (cluster eqiad1) * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-8 (cluster eqiad1) * 12:05 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:56 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:24 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:19 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.instance.stop_start (exit_code=97) vm tools-legaci-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legaci-redirector-3 (cluster eqiad1) === 2025-11-17 === * 18:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T409981|T409981]]) * 10:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T409981|T409981]]) === 2025-11-14 === * 16:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-4 ([[phab:T409287|T409287]]) * 16:26 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-4 ([[phab:T409287|T409287]]) === 2025-11-13 === * 15:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-bastion-14 (cluster eqiad1) * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-bastion-14 (cluster eqiad1) * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-8 (cluster eqiad1) === 2025-11-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-11 === * 15:28 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 ([[phab:T399313|T399313]]) === 2025-11-10 === * 22:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 22:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:16 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:14 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:08 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest:latest * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-07 === * 11:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:35 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 ([[phab:T409287|T409287]]) * 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 ([[phab:T409287|T409287]]) === 2025-11-06 === * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 taavi: taavi@tools-bastion-15:~ $ sudo loginctl terminate-user damian * 14:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:53 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) === 2025-11-04 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 17:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-10-30 === * 18:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:39 taavi: kick off script to rebuild all pre-built images, including [[phab:T407707|T407707]] * 16:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 16:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 16:27 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T408669|T408669]]) * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6 * 12:48 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6 * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:07 taavi: delete paws, paws-master security groups, long obsolete as paws is now in a separae project * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-10-27 === * 22:09 taavi: copy toolviews database hiera data to a place where haproxy nodes can see them [[phab:T408454|T408454]] * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:16 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-24 === * 20:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t * 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k * 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106 * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106 * 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103 * 13:37 andrewbogott: rebooting clouddumps100[12] for [[phab:T407110|T407110]] === 2025-10-23 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:11 taavi: deleting old nginx front proxy instances [[phab:T283948|T283948]] * 10:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 15:56 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3 * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:35 taavi: moving toolforge traffic to haproxy directly [[phab:T283948|T283948]] * 07:00 godog: delete tools-nfs-2 - [[phab:T404584|T404584]] === 2025-10-21 === * 08:53 godog: shut down tools-nfs-2 - [[phab:T404584|T404584]] * 07:52 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 07:49 godog: resize tools-nfs-3 to match tools-nfs-2 (g4.cores16.ram64.disk20.10xiops) - [[phab:T404584|T404584]] * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-20 === * 16:31 taavi: make logrotate run hourly on haproxy nodes [[phab:T284558|T284558]] === 2025-10-16 === * 12:01 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-15 === * 08:03 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 08:00 godog: resize tools-nfs-3 - [[phab:T404584|T404584]] === 2025-10-14 === * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:45 godog: update nfs-tools.wmcloud.org and nfs.svc.toolforge.org proxied to point to tools-nfs-3 === 2025-10-13 === * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:17 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, too * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:13 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:13 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:10 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:01 godog: switch NFS from tools-nfs-2 to tools-nfs-3 - [[phab:T404584|T404584]] * 07:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81 === 2025-10-10 === * 09:22 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 * 09:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 === 2025-10-09 === * 08:21 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 08:15 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2025-10-08 === * 21:47 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 21:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 12:27 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]] * 10:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 06:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 * 06:43 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 === 2025-10-07 === * 18:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11 * 15:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69 * 15:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69 * 14:51 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 14:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:58 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-06 === * 12:06 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7 * 11:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7 * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 08:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:18 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 * 07:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 === 2025-10-03 === * 12:51 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-6 * 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-6 * 12:45 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-5 * 09:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-5 === 2025-10-02 === * 13:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 09:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 * 08:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 === 2025-10-01 === * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-30 === * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 * 08:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 === 2025-09-29 === * 13:23 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 13:23 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 11:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 07:00 godog: kick stuck nfs workers from clouddumps1001 === 2025-09-28 === * 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:13 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:07 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:07 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-25 === * 18:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 20:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:30 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:07 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 13:49 dcaro: patched all tools with new resource defaults, everything looks good * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro: depolyed jobs-api change to default resources, patching existing jobs * 13:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 03:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 === 2025-09-23 === * 20:08 andrewbogott: creating puppetdbpostgres and adding it to tools-puppetdb-2 to store postgres data; the root volume of that VM was filling up and causing widespread puppet issues * 01:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-09-22 === * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-21 === * 09:17 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 03:16 dcaro: acking and silencing CPU capacity alerts to handle on Monday, they should not page * 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 01:46 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-113.tools.eqiad1.wikimedia.cloud to the cluster * 01:36 andrewbogott: adding additional worker node in response to repeated capacity alerts * 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-09-19 === * 13:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 13:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 === 2025-09-18 === * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:52 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 08:34 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 06:47 taavi: delete tools-sgebastion-10 [[phab:T314665|T314665]] === 2025-09-17 === * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 12:53 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 * 08:08 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 === 2025-09-16 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:57 taavi: delete tools-sgebastion puppet prefix [[phab:T314665|T314665]] * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 taavi: shutdown tools-sgebastion-10 [[phab:T314665|T314665]] * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-13 * 14:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-13 * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-12 * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-12 * 07:11 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 06:51 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-15 === * 16:22 taavi: reboot old bastions to kick long-living connections into newer ones * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:47 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 * 12:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 === 2025-09-12 === * 08:49 taavi: pointing login.toolforge.org to tools-bastion-15 [[phab:T392510|T392510]] * 08:33 taavi: pointing dev.toolforge.org to tools-bastion-14 [[phab:T392510|T392510]] * 07:14 godog: uncordon tools-k8s-worker-nfs-53 after failed cookbook (?) yesterday === 2025-09-11 === * 14:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:36 godog: drain/reboot tools-k8s-worker-nfs-46 - [[phab:T404322|T404322]] * 14:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:22 andrewbogott: actually I didn't drain tools-k8s-worker-nfs-53 because the alert cleared on its own * 14:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-53 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 14:21 andrewbogott: draining/rebooting tools-k8s-worker-nfs-53 because of procs in D state * 13:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 13:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-09-10 === * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:45 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) === 2025-09-09 === * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:55 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) ([[phab:T404047|T404047]]) * 08:55 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot ([[phab:T404047|T404047]]) === 2025-09-08 === * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:16 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:40 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests === 2025-09-06 === * 23:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 * 23:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 === 2025-09-05 === * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 19:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:52 dcaro: added 'disable-ssl' to tools replica.my.cnf === 2025-09-03 === * 17:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 17:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-29 === * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting === 2025-08-28 === * 16:52 taavi: rebuild tcl, mariadb images on top of trixie [[phab:T400256|T400256]] * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:05 taavi: copy missing aptly packages to trixie-<nowiki>{</nowiki>tools,toolsbeta<nowiki>}</nowiki> * 11:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-26 === * 13:42 dcaro: extended object storage quota to 100G ([[phab:T402923|T402923]]) * 10:25 dhinus: shut down tools-harbor-1 (no longer used) === 2025-08-25 === * 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-81 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-81 === 2025-08-21 === * 12:28 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:31 godog: reboot nfs workers to reset processes stuck in D state * 07:28 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 04:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 * 03:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 === 2025-08-20 === * 08:09 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) === 2025-08-19 === * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:37 dcaro: flipped the tools-harbor.wmcloud.org endpoint to point to tools-harbor-2 ([[phab:T350687|T350687]]) * 14:22 Raymond_Ndibe: setting tools-harbor-1 as read-only ([[phab:T350687|T350687]]) * 13:24 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 13:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.16.0 ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/metrics-server:v0.7.2 ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud === 2025-08-16 === * 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-111 * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-111 * 21:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-111 * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-111 === 2025-08-15 === * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 19:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 * 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 === 2025-08-14 === * 15:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 11:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 02:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:36 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) * 11:18 taavi: delete tools-prometheus-6, shutdown for a while * 08:51 godog: bounce stashbot * 08:33 godog: refresh machine-id on tools-k8s-worker-[102-103,105-112].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-3,5,7-14,16-17,19,21-24,26-27,32-48,50,53-55 ,57-58,61,65-82].tools.eqiad1.wikimedia.cloud === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:34 taavi: building initial trixie based images [[phab:T400255|T400255]] * 12:50 dcaro: redepoly kyverno ([[phab:T394787|T394787]]) * 12:49 dcaro: manually migrate cleanuppolicies.kyverno.io and clustercleanuppolicies.kyverno.io (using kyverno cli) ([[phab:T394787|T394787]]) * 10:01 dcaro: starting upgrade for kyverno ([[phab:T394787|T394787]]) * 10:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 === 2025-08-08 === * 06:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 * 14:20 andrewbogott: draining and rebooting tools-k8s-worker-nfs-67 * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:54 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:41 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:39 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:38 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-05 === * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 03:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 02:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 * 11:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 === 2025-08-01 === * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 04:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-07-30 === * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 15:56 andrewbogott: draining and restarting tools-k8s-worker-nfs-74 * 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrewbogott: draining and restarting tools-k8s-worker-nfs-58 and tools-k8s-worker-nfs-32 * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:16 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-82.tools.eqiad1.wikimedia.cloud to the cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-81.tools.eqiad1.wikimedia.cloud to the cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.quota_increase * 12:39 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-80.tools.eqiad1.wikimedia.cloud to the cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:00 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97) * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:28 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:07 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:59 wmbot~dcaro@acme: Added a new k8s worker tools-k8s-worker-112.tools.eqiad1.wikimedia.cloud to the cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:14 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-07-28 === * 20:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:58 taavi: update pywikibot image to 10.2.0 [[phab:T396933|T396933]] === 2025-07-26 === * 07:16 wmbot~root@toolforge: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:16 wmbot~root@toolforge: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-23 === * 18:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-21 === * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-07-19 === * 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 13:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-07-18 === * 10:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 === 2025-07-14 === * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78 === 2025-07-13 === * 03:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 * 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 === 2025-07-11 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 * 09:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 === 2025-07-09 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:55 dcaro: adding arch arm64 to all toolforge repos ([[phab:T398016|T398016]]) * 09:40 dcaro: added arch arm64 to jessie-tools repo ([[phab:T398016|T398016]]) === 2025-07-08 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-07 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 17:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 16:39 dcaro: pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest * 16:05 dcaro: clearing images from tools-imagebuilder-2 as it's out of space * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-06 === * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 * 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 === 2025-07-05 === * 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs === 2025-07-04 === * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-07-03 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-02 === * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55 * 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-06-30 === * 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) === 2025-06-28 === * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:08 dcaro: left a tmux running with a script to restart nginx if stuck * 09:59 dcaro: restarted nginx in tools-static === 2025-06-27 === * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 === 2025-06-26 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 * 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 === 2025-06-24 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-23 === * 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]] === 2025-06-22 === * 00:09 andrewbogott: rebooting tools-prometheus-8 === 2025-06-21 === * 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of D state * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-19 === * 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools === 2025-06-18 === * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable === 2025-06-16 === * 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 === 2025-06-14 === * 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-06-12 === * 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh) * 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-11 === * 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 === 2025-06-10 === * 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved === 2025-06-06 === * 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark * 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-06-05 === * 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert * 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-30 === * 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 07:38 taavi: reboot tools-static-15 to unstuck NFS things === 2025-05-24 === * 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 * 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 === 2025-05-23 === * 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 * 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 === 2025-05-22 === * 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:15 dcaro: restart tools-static nginx due to nfs hiccup * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7 * 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 * 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]] === 2025-05-21 === * 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh) * 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead === 2025-05-20 === * 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97) * 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown * 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-19 === * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-14 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36 * 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-05-12 === * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512 ([[phab:T393686|T393686]]) * 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 * 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 === 2025-05-10 === * 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo<nowiki>{</nowiki>,.socket<nowiki>}</nowiki> # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]]) * 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-<nowiki>{</nowiki>pam,sudo<nowiki>}</nowiki>.service && systemctl restart sssd-pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket # try to reset the rate limits this way ([[phab:T393732|T393732]]) * 16:22 lucaswerkmeister: systemctl restart sssd-<nowiki>{</nowiki>pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>,sudo<nowiki>}</nowiki>.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient) * 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam<nowiki>{</nowiki>,<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket<nowiki>}</nowiki> # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]? === 2025-05-09 === * 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake * 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory) * 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs === 2025-05-08 === * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state * 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-07 === * 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2 * 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2 * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:33 dcaro: released jobs-cli 16.1.12 * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 === 2025-05-04 === * 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06) === 2025-05-02 === * 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]] * 13:55 taavi: reboot tools-static-15 === 2025-04-28 === * 13:07 dhinus: tools-db-4: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:06 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:05 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T318479|T318479]] === 2025-04-24 === * 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12 * 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing "sudo: PAM account management error: Authentication service cannot retrieve authentication info" * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:51 taavi: add missing ICMPv6 security group rule to 'default' group * 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]] === 2025-04-23 === * 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles === 2025-04-21 === * 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]] === 2025-04-17 === * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-11 === * 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-10 === * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2025-04-09 === * 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff) * 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-04-07 === * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-06 === * 02:12 andrewbogott: truncating large logfiles on tools nfs === 2025-04-04 === * 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes === 2025-04-03 === * 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 === 2025-04-02 === * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 === 2025-04-01 === * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 === 2025-03-31 === * 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 * 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 === 2025-03-28 === * 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]]) * 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]]) * 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]]) * 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]]) === 2025-03-27 === * 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster * 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 === 2025-03-25 === * 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 * 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2 * 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 * 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-03-22 === * 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-18 === * 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) * 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) === 2025-03-17 === * 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) === 2025-03-16 === * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 === 2025-03-15 === * 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]] === 2025-03-13 === * 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 === 2025-03-12 === * 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o * 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-03-07 === * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-03-06 === * 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-03-05 === * 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values) * 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes * 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-03-04 === * 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual) * 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-03 === * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-01 === * 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 === 2025-02-27 === * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-02-26 === * 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-25 === * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 === 2025-02-24 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-21 === * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-02-20 === * 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-19 === * 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-02-18 === * 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]]) * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]]) === 2025-02-17 === * 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-10 === * 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-02-09 === * 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes === 2025-02-07 === * 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]] * 17:58 andrewbogott: "SET GLOBAL read_only=OFF; " on tools-db-4; both -5 and -4 were set to read only. No idea why or how... * 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 === 2025-02-06 === * 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage * 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-03 === * 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 === 2025-02-01 === * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102 * 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]] * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102 * 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]] * 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 * 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 === 2025-01-31 === * 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]] === 2025-01-29 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-27 === * 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-26 === * 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster * 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster * 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 * 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 === 2025-01-24 === * 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 === 2025-01-23 === * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs === 2025-01-22 === * 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2025-01-18 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2025-01-17 === * 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing) === 2025-01-15 === * 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-13 === * 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:49 dcaro: restart prometheus to pick up the new ips for vms and such * 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 * 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 * 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots === 2025-01-12 === * 09:53 taavi: hard reboot tools-k8s-worker-nfs-55 === 2025-01-08 === * 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22 * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22 * 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70 * 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70 * 14:16 dcaro: reboot tools-static-15 nfs is stuck === 2025-01-07 === * 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor === 2025-01-06 === * 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-01-05 === * 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13 === 2025-01-03 === * 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 * 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69 * 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69 === 2025-01-02 === * 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 === 2025-01-01 === * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 === 2024-12-13 === * 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 * 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 === 2024-12-12 === * 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2024-12-06 === * 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]]) * 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]]) * 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing * 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main === 2024-11-29 === * 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-27 === * 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns # update resolv.conf in coredns containers === 2024-11-26 === * 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:22 dcaro: rebooting k8s-control-9 * 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:17 dcaro: rebooting k8s-control-8 * 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:14 dcaro: restarting tools-k8s-worker-nfs-72 * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 09:12 dcaro: restarting tools-k8s-worker-nfs-70 * 09:11 dcaro: restarting tools-k8s-worker-nfs-50 * 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 * 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]]) * 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]]) === 2024-11-25 === * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2024-11-23 === * 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]]) * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]]) * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-18 === * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-15 === * 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]]) * 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) === 2024-11-14 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-10 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]]) === 2024-11-06 === * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 dcaro: added api.svc.toolforge.org dns record entry * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-22 === * 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 * 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]] === 2024-10-16 === * 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-14 === * 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]]) * 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 * 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-10-09 === * 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup * 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating * 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]] * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-04 === * 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-02 === * 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-01 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:28 dcaro: updated ci image with latest precommit versions * 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-30 === * 18:25 taavi: run striker migrations [[phab:T359428|T359428]] === 2024-09-28 === * 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2024-09-24 === * 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-20 === * 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]]) * 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network * 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-17 === * 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64 * 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63 * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62 * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60 * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6 * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56 * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49 * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31 * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30 * 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29 * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28 * 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster * 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]]) === 2024-09-16 === * 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-09-13 === * 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) === 2024-09-12 === * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) * 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) === 2024-09-11 === * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-09 === * 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager === 2024-09-06 === * 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6 * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-09-05 === * 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-03 === * 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 * 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 === 2024-09-02 === * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15 * 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15 * 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2 * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster * 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-08-26 === * 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster * 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster * 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 11:06 dcaro: manually deleted the coredns pods that had been around for 4d * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:18 dcaro: scale up cordens deployment to 4 replicas === 2024-08-21 === * 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-19 === * 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 === 2024-08-15 === * 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 * 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 === 2024-08-13 === * 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-08-12 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-06 === * 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2024-08-05 === * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot * 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected. * 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55 * 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc * 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]] * 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 * 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 === 2024-07-30 === * 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2024-07-29 === * 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) * 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance * 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission * 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 17:00 dcaro: moving the toolforge apt repo to tools-services-06 * 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud * 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-19 === * 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6 * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-07-18 === * 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16 * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16 * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-11 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1 * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0 * 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]] * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]]) * 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]]) * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23 * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23 * 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1 * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24 * 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24 * 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22 * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22 * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]]) * 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]] * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 === 2024-06-25 === * 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-> dict[str, Any]` type annotations causing [[phab:T368463|T368463]] * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104 * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103 * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56 * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55 * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54 * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53 * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52 * 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51 * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53 * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52 * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7 * 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7 * 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8' * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8' * 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9 * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9 * 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47 * 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44 * 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44 * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42 * 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41 * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39 * 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38 * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37 * 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36 * 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35 * 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35 * 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34 * 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33 * 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35 * 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33 * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33 * 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33 === 2024-06-24 === * 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots. * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32 * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29 * 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29 * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28 * 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28 * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27 * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27 * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10 * 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10 * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13 * 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13 * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26 * 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22 * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22 * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21 * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21 * 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19 * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20 * 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17 * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14 * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13 * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13 * 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12 * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12 * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7 * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8 * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8 * 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8 * 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15 * 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4 * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4 * 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10 * 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10 * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9 * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7 * 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7 * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3 * 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5 * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7 * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7 * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7 * 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43 * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7 * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7 * 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7 * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43 * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7 * 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6 * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6 * 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1 * 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5 * 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5 * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1 * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6 * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2 * 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2 * 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2 * 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2 * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2 * 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05 * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8 * 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8 * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5 * 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5 * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-06-20 === * 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]] * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-14 === * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]]. * 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 13:45 taavi: hard reboot tools-k8s-control-7 * 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:57 dcaro: cleaning old maintain-kubeusers configmaps * 10:45 dcaro: cleaning up old resourcequotas === 2024-06-10 === * 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno === 2024-06-07 === * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 dcaro: deploying toolforge-webservice 0.103.6 * 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13 * 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12 === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-05-29 === * 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api * 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-28 === * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-25 === * 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-22 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-15 === * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO * 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9 * 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 === 2024-05-07 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-05 === * 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]] === 2024-04-24 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-17 === * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13 * 10:48 dcaro: rebooting tools-k8s-worker-nfs-1 === 2024-04-16 === * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0' * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0' === 2024-04-15 === * 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-11 === * 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-09 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission * 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6 === 2024-04-08 === * 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 * 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5 === 2024-04-05 === * 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]] === 2024-04-02 === * 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]]) * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05 === 2024-03-28 === * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05 * 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]] * 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06 * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06 * 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' === 2024-03-27 === * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01 === 2024-03-26 === * 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion' * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion' * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11 * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11 * 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]] === 2024-03-25 === * 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector * 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud === 2024-03-22 === * 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error) === 2024-03-21 === * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4 * 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3 * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04 * 12:30 taavi: move checker service address to tools-checker-5 * 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker' * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' === 2024-03-19 === * 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images) * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14 * 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]] * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]]) === 2024-03-18 === * 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 taavi: restart harbor services after docker service restart * 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1 * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1 * 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]] * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]] === 2024-03-15 === * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48' * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48' * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01 * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]] * 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]] === 2024-03-13 === * 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable. === 2024-03-12 === * 12:38 taavi: hard reboot tools-prometheus-6 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-11 === * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]] === 2024-03-09 === * 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs === 2024-03-08 === * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-07 === * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-06 === * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates * 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs === 2024-03-05 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]]) * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud === 2024-03-04 === * 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state === 2024-03-03 === * 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state) === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-29 === * 14:36 dcaro: deploy webservice 0.103.3 === 2024-02-28 === * 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]]) * 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) === 2024-02-23 === * 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2 * 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]] === 2024-02-22 === * 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster * 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 === 2024-02-21 === * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4 * 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster === 2024-02-20 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster * 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101 * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102 * 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102 * 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster * 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100 * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster * 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96 * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95 * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95 * 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92 * 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster * 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91 * 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91 * 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90 * 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88 === 2024-02-19 === * 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster * 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84 * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83 * 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81 * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81 * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-16 === * 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80 * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77 * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77 === 2024-02-15 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4 * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4 * 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76 * 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75 * 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster === 2024-02-14 === * 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30 * 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3 * 16:30 taavi: reboot tools-sgeexec-10-23 due to high load * 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74 * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster * 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71 * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71 * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster * 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70 * 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70 * 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69 * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68 * 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68 === 2024-02-13 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster * 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65 * 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64 * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64 === 2024-02-12 === * 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster * 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster * 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61 * 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60 * 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60 * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster * 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58 * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58 * 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54 * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15 * 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-11 === * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-02-09 === * 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule * 13:06 taavi: reboot tools-sgecron-2 due to high load * 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config * 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 * 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes === 2024-02-08 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 * 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42 * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42 === 2024-02-07 === * 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers * 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers === 2024-02-06 === * 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-01-31 === * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-30 === * 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster * 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster * 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster * 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41 * 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41 * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40 * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40 * 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39 * 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39 * 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38 * 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38 * 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37 * 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37 * 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]]) * 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]]) * 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]] * 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]]) * 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]]) * 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]]) === 2024-01-29 === * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud * 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud * 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster * 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33 * 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33 * 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32 * 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32 * 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31 * 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30 * 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster * 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB * 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB * 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-26 === * 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta === 2024-01-25 === * 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1 === 2024-01-23 === * 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22 * 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-18 === * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17 === 2024-01-17 === * 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]] * 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]]) * 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]]) * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]] === 2024-01-15 === * 09:18 taavi: reboot stuck tools-k8s-worker-84 === 2024-01-12 === * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12' * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12' === 2024-01-11 === * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-10 === * 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:17 taavi: reboot tools-k8s-worker-98 === 2024-01-09 === * 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor -- [[phab:T354714|T354714]] * 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it) * 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder * 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder * 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:13 taavi: reboot tools-sgeexec-10-17 due to high load === 2024-01-08 === * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28 * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 taavi: reboot tools-sgeexec-10-21 === 2024-01-05 === * 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-04 === * 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3 === 2024-01-03 === * 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs * 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage === 2024-01-02 === * 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]]) * 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]]) * 10:37 dcaro: hard reboot tools-harbor-1 * 10:13 dhinus: hard reboot tools-harbor-1 === 2024-01-01 === * 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]] ==Archives== * [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014) * [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017) * [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019) * [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021) * [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023) </noinclude> {{SAL|Project Name=tools}} <noinclude>[[Category:SAL]]</noinclude> f4u1qz8s5uw7dtv67q8axwgn70ty7g7 498 6726 2466284 2465799 2026-10-11T07:33:34Z Stashbot 7414 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2466284 wikitext text/x-wiki === 2026-10-11 === * 07:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api === 2026-10-08 === * 16:56 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-07 === * 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T440326|T440326]]) * 13:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T440326|T440326]]) * 13:55 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api ([[phab:T440326|T440326]]) * 13:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T440326|T440326]]) * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-10-06 === * 17:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2026-10-05 === * 18:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 17:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 17:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 12:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 12:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-10-02 === * 11:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 11:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) === 2026-10-01 === * 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-30 === * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 09:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 08:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:07 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 07:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-29 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-24 === * 09:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component alerts-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-23 === * 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:56 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:45 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:44 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:34 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:33 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:26 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-22 === * 17:17 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 17:12 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 17:09 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 17:04 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:52 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 16:47 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:44 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 16:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:31 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 16:26 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 07:35 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 07:24 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 07:22 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 07:15 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-21 === * 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-backup-1.toolsbeta.eqiad1.wikimedia.cloud * 14:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-backup-1.toolsbeta.eqiad1.wikimedia.cloud * 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 10:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-09-18 === * 20:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 14:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 14:19 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 14:19 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) * 13:37 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api ([[phab:T432865|T432865]]) * 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api ([[phab:T432865|T432865]]) === 2026-09-15 === * 17:47 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli * 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-09-14 === * 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 22:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 22:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 22:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 22:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 22:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 22:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 22:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-cli * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli * 15:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-cli * 15:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-cli === 2026-09-11 === * 08:57 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:46 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-09-10 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:31 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-emailer * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-09-07 === * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 14:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli === 2026-09-04 === * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component alerts-api * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component alerts-api === 2026-08-30 === * 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 10:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2026-08-26 === * 09:49 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:14 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:00 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:59 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:54 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 08:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-20 === * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:06 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 16:33 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 15:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:52 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 11:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 11:38 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 11:38 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2026-08-13 === * 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 08:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 02:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) * 02:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) * 02:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) * 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) * 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) * 02:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) * 01:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13 * 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-13 * 01:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-13 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-13 * 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 01:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) === 2026-08-12 === * 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T433131|T433131]]) * 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T433131|T433131]]) * 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 19:05 andrew@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 18:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) * 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) * 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 16:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 16:02 andrewbogott: upgrading to kubernetes 1.33.13 * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.33.13 ([[phab:T433131|T433131]]) * 15:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.33.13 ([[phab:T433131|T433131]]) * 15:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-bastion-6 ([[phab:T401804|T401804]]) * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-bastion-6 ([[phab:T401804|T401804]]) * 12:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 14:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 14:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-05 === * 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 14:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-04 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:05 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:51 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 09:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:47 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-03 === * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-harbor-1 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-harbor-1 * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:25 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 08:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 08:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-31 === * 12:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:53 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:44 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:19 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 09:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission === 2026-07-27 === * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:14 godog: set 'dumps_use_nfs_lb: true' for prefix-puppet toolsbeta-test-k8s-worker-nfs - [[phab:T432325|T432325]] * 10:30 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T432325|T432325]]) * 10:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 09:32 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 09:29 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) === 2026-07-23 === * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-20 === * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:29 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:20 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:19 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:19 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-16 === * 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:49 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 15:07 godog: fail back to classic dumps nfs for toolsbeta-test-k8s-worker-nfs prefix * 12:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-harbor' * 12:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-harbor' * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-6 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-6 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-5 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-5 * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-4 * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-4 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-cumin-1 * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-cumin-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-cumin' * 10:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-cumin' * 08:49 godog: set 'dumps_use_nfs_lb: true' for mail, bastion, static prefixes * 08:22 godog: test reboot all workers * 07:58 godog: test dumps-nfs switchover on toolsbeta-test-k8s-worker-nfs-8 * 07:44 godog: set 'dumps_use_nfs_lb: true' for toolsbeta-test-k8s-worker-nfs prefix * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 21:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:25 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud === 2026-07-09 === * 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 09:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 09:35 godog: carry out nfs tests on toolsbeta-test-k8s-worker-nfs-8 - [[phab:T411248|T411248]] * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 16:37 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 16:28 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 16:28 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-06-24 === * 16:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:30 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-ap * 12:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-ap === 2026-06-15 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2026-06-11 === * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-08 === * 10:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T427918|T427918]]) * 10:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T427918|T427918]]) * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:03 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T427918|T427918]]) * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T427918|T427918]]) * 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:14 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) === 2026-06-04 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-06-02 === * 15:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 01:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 19:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 07:54 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 07:52 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-05-21 === * 09:48 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:33 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:29 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:13 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:09 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2026-05-20 === * 23:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 23:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 23:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 23:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:27 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 08:26 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 08:26 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 08:26 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 00:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 23:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 23:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 09:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 09:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-14 === * 15:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway ([[phab:T425479|T425479]]) * 10:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway ([[phab:T425479|T425479]]) === 2026-05-13 === * 17:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:19 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 10:36 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 10:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-11 === * 17:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:23 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 15:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 10:35 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 10:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 00:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-06 === * 08:57 taavi: renew prometheus kubernetes certificate about to expire * 08:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-05-05 === * 04:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:49 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-emailer * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 19:43 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 10:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node === 2026-04-21 === * 14:10 taavi: delete all ingress objects from toolsbeta [[phab:T392356|T392356]] * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-10 === * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-04-07 === * 23:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 13:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 13:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 07:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 07:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 10:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:34 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 17:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:20 godog: reboot harbordb1 instance * 12:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-26 === * 17:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-24 === * 17:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 14:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2026-03-17 === * 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node === 2026-03-10 === * 18:20 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 11:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice-cli * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice-cli * 15:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:53 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:42 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 65536 ram * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 65536 ram * 12:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:22 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-gateway-1 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-gateway-1 * 11:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster * 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster === 2026-03-06 === * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2026-03-05 === * 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:02 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 14:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:44 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:11 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-04 === * 21:35 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:19 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:40 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:30 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:21 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:20 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:20 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:19 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:19 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:18 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:35 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:34 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:21 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:20 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 04:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:48 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:48 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:47 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:47 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:46 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 04:41 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 04:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 04:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 04:32 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 04:15 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 04:14 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:04 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:32 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:31 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:09 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:47 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:43 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:53 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 01:19 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 01:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:53 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:45 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:40 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:39 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:37 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:27 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:25 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 00:04 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) === 2026-03-03 === * 23:55 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:54 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:53 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:51 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:51 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:49 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 23:01 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:55 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:47 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:41 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:40 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:40 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:29 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:16 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:15 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 22:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:01 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:56 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 21:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:27 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:17 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:15 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 20:51 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 20:50 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 20:01 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:41 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:19 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-02-25 === * 14:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component gateway-api * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-24 === * 10:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-16 === * 09:10 dcaro: restarted the admin tool, was giving error connecting to lighttpd socket === 2026-02-05 === * 18:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2026-02-04 === * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-02-03 === * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-01-28 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-12 === * 17:59 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 17:55 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission * 17:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 17:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-12, toolsbeta-test-k8s-ingress-9 * 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-12, toolsbeta-test-k8s-ingress-9 * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 13:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-10 ([[phab:T413796|T413796]]) * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-10 ([[phab:T413796|T413796]]) * 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-11 ([[phab:T413796|T413796]]) * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-11 ([[phab:T413796|T413796]]) * 13:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 13:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 02:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:13 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 01:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 00:56 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 00:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 100k * 11:06 godog: bump object quota to 200G === 2025-12-15 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics === 2025-12-11 === * 16:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 18:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 15:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 15:41 andrewbogott: deleting toolsbeta-test-k8s-etcd-27 and replacing with a Bullseye node for cluster consistency [[phab:T361237|T361237]] === 2025-12-03 === * 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 14:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T375217|T375217]]) * 14:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T375217|T375217]]) * 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T375217|T375217]]) * 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T375217|T375217]]) === 2025-12-02 === * 22:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 22:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster === 2025-12-01 === * 16:22 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-25 === * 13:27 volans: temporarily disabling puppet on the nfs workers to merge gerrit/1210582 * 10:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-prometheus-2 (cluster eqiad1) * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-prometheus-2 (cluster eqiad1) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:03 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 09:47 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 14:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:06 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 11:27 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:27 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 09:47 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-19 === * 16:50 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:33 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component infra-tracing * 16:18 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:11 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component infra-tracing * 16:11 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:10 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:46 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-18 === * 17:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:59 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-11-17 === * 18:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 18:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-13 === * 15:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-bastion-7 (cluster eqiad1) * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-bastion-7 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-haproxy-8 (cluster eqiad1) * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-haproxy-7 (cluster eqiad1) * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-haproxy-7 (cluster eqiad1) === 2025-11-12 === * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-10 === * 22:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 20:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-06 === * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-04 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:43 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 00:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 20:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 10:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 10:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 10:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-11-01 === * 00:06 bd808: Re-enabled Puppet on toolsbeta-test-k8s-haproxy-7 ([[phab:T408570|T408570]]) === 2025-10-31 === * 19:53 bd808: Disabled Puppet on toolsbeta-test-k8s-haproxy-7 for manual testing of haproxy changes ([[phab:T408570|T408570]]) === 2025-10-30 === * 18:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-10-27 === * 18:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:10 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-23 === * 17:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 17:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 17:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 17:08 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 16:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-10-21 === * 11:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 11:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 00:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-16 === * 11:20 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:12 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-14 === * 14:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-13 === * 14:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-08 === * 11:40 godog: nfs back up and has been for some time [[phab:T406688|T406688]] * 10:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:34 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:19 godog: shut down nfs while investigating [[phab:T406688|T406688]] * 09:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-07 === * 18:56 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 18:56 root@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:56 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) * 11:52 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 11:46 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 11:44 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 11:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:56 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 08:52 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 08:39 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:39 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:37 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:36 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:35 godog: move nfs from toolsbeta-nfs-4 to toolsbeta-nfs-5 - [[phab:T404584|T404584]] * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 07:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-10-03 === * 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-6 * 12:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-5 * 09:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-5 === 2025-10-02 === * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) === 2025-10-01 === * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-29 === * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found === 2025-09-26 === * 08:17 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 08:07 wmbot~godog@r5: START - Cookbook wmcs.nfs.add_server === 2025-09-25 === * 17:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:39 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:41 godog: test reboot of toolsbeta-nfs-4 * 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 17:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:31 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-11 * 09:19 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-11 * 09:17 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 09:11 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 09:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9 * 09:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-test-k8s-worker-nfs-9,toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9,toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-nfs-7 * 09:03 filippo@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-14 * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 08:42 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-bastion-7 * 08:41 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-bastion-7 * 08:19 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:19 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:03 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers * 07:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 06:34 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 06:26 wmbot~godog@r5: START - Cookbook wmcs.nfs.add_server === 2025-09-23 === * 15:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-09-22 === * 16:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-18 === * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-09-16 === * 16:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:56 taavi: delete toolsbeta-sgebastion puppet prefix * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:43 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-ingress-10 ([[phab:T404721|T404721]]) * 15:42 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-ingress-10 ([[phab:T404721|T404721]]) * 15:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 15:35 wmbot~dcaro@acme: Added a new k8s ingress toolsbeta-test-k8s-ingress-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:24 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 14:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:24 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:24 wmbot~dcaro@acme: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 14:11 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-worker-nfs-11 ([[phab:T404721|T404721]]) * 14:09 wmbot~dcaro@acme: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-worker-nfs-11 ([[phab:T404721|T404721]]) * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 13:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 13:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-5 * 13:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-5 * 13:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm toolsbeta-test-k8s-worker-nfs-5 (cluster eqiad1, project toolsbeta) * 13:31 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm toolsbeta-test-k8s-worker-nfs-5 (cluster eqiad1, project toolsbeta) * 13:22 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5 * 13:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5 * 13:21 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-test-k8s-worker-nfs-5 * 13:20 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-test-k8s-worker-nfs-5 * 13:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-15 === * 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 11:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 11:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-09-10 === * 14:43 godog: reset puppet to production, testing finished * 14:39 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1186938 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1186937 * 08:40 volans: add volans to toolsbeta.admin tool * 08:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'volans' in role 'member' * 08:15 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'volans' in role 'member' === 2025-09-09 === * 13:22 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm toolsbeta-cumin-1 (cluster eqiad1, project toolsbeta) * 13:22 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm toolsbeta-cumin-1 (cluster eqiad1, project toolsbeta) * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component mainain-kubeusers * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component mainain-kubeusers === 2025-09-08 === * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2025-09-05 === * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro: bumped the object quota to 100G, like tools === 2025-09-03 === * 17:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 09:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 23:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-01 === * 21:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 21:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 21:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 19:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 19:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 19:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 19:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:26 dcaro: upgrade finished * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:47 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=97) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:59 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 12:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) * 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:26 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:26 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.run_tests * 08:06 dcaro: starting upgrade to v1.30 ([[phab:T402377|T402377]]) === 2025-08-28 === * 08:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 18:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 17:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:38 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-21 === * 10:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:21 dcaro: test for sal === 2025-08-19 === * 16:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 20:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-14 === * 15:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:23 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) , no stuck workers found * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers , no stuck workers found * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for * 09:50 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for * 09:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for * 03:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 03:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 03:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 03:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 03:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:14 dcaro: deploying kyverno ([[phab:T394787|T394787]]) === 2025-08-08 === * 06:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 06:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 05:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:34 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli ([[phab:T401014|T401014]]) * 17:33 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:21 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 17:21 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 17:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2025-08-05 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 12:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 12:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tool-webservice * 12:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tool-webservice * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 08:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 02:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 02:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 01:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 01:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 01:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 01:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 01:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 01:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 00:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) === 2025-08-01 === * 03:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 03:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 15:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 03:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 03:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 03:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-30 === * 09:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-28 === * 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-07-23 === * 18:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-14 === * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398715|T398715]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398715|T398715]]) === 2025-07-10 === * 10:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 09:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 09:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-07-09 === * 17:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:07 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 08:46 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component misctools-cli * 08:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 08:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 08:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli === 2025-07-08 === * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-07-07 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-03 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-02 === * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers * 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-06-26 === * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-24 === * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging * 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-06-23 === * 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-19 === * 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-06-18 === * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-06-17 === * 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-16 === * 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-12 === * 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-11 === * 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-10 === * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]]) * 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]]) === 2025-06-09 === * 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-07 === * 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G === 2025-06-06 === * 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-06-05 === * 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-06-04 === * 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-02 === * 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-22 === * 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6 * 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5 * 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5 * 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 === 2025-05-21 === * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud * 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-20 === * 18:24 bd808: Made addshore an admin === 2025-05-19 === * 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-15 === * 08:13 taavi: renew expiring Puppet CA cert === 2025-05-14 === * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-12 === * 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-05-09 === * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-08 === * 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-07 === * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-04-24 === * 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-04-23 === * 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-21 === * 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]] === 2025-04-17 === * 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]] === 2025-04-11 === * 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-09 === * 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-04-07 === * 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-04 === * 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-31 === * 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) === 2025-03-25 === * 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-13 === * 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-12 === * 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder * 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-06 === * 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-05 === * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-04 === * 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission === 2025-03-03 === * 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-02-27 === * 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-02-26 === * 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-02-24 === * 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-19 === * 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-17 === * 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-06 === * 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-01 === * 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]] * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes === 2025-01-29 === * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-23 === * 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]]) * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]]) * 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-22 === * 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-01-21 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9 * 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9 * 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8 * 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8 * 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5 * 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5 * 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10 * 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers * 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10 === 2025-01-20 === * 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-01-17 === * 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-01-15 === * 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-07 === * 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics * 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics * 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics * 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-01-06 === * 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2024-12-13 === * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-12-06 === * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-29 === * 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-25 === * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-11-23 === * 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]]) * 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]]) * 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice * 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice === 2024-11-18 === * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-14 === * 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-07 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-11-06 === * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-30 === * 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) === 2024-10-29 === * 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1 * 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1 === 2024-10-16 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-10 === * 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-10-09 === * 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]]) * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers * 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers === 2024-10-04 === * 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-03 === * 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]] * 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) === 2024-10-01 === * 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-28 === * 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]]) * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]]) * 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10 * 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7 * 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7 * 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10 * 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10 * 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10 * 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10 * 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster * 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-24 === * 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-21 === * 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 === 2024-09-20 === * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]]) * 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]]) * 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-11 === * 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster === 2024-09-10 === * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) === 2024-09-09 === * 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) === 2024-09-06 === * 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-09-05 === * 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9 * 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9 * 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8 * 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7 * 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7 * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-03 === * 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno * 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2024-09-02 === * 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2024-08-28 === * 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back. * 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15 * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15 * 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15 * 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15 * 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15 * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico === 2024-08-26 === * 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-21 === * 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests * 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-13 === * 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-08-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components * 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents === 2024-08-06 === * 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-05 === * 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1) * 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-30 === * 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli === 2024-07-29 === * 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli * 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli * 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli * 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli * 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-18 === * 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api * 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-12 === * 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16 * 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16 * 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16 * 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16 === 2024-07-11 === * 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16 * 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16 * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16 * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16 * 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16 * 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16 * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16 * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16 * 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16 * 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16 * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16 * 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]] * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16 * 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16 * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers * 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway * 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26 * 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26 * 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25 * 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25 * 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23 * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23 * 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23 * 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]]) * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-25 === * 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5' * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5' * 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6' * 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-24 === * 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-21 === * 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd === 2024-06-20 === * 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-17 === * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7 * 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10 * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10 * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5 * 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5 * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2 * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1 * 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6 * 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5 * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5 * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1 * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6 * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6 * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2 * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2 * 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2 * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2 === 2024-06-14 === * 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1 * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1 * 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-07 === * 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-05-30 === * 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-29 === * 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-28 === * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-25 === * 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-15 === * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-05-10 === * 13:57 taavi: renew k8s prometheus certificate === 2024-05-07 === * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-04 === * 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json * 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-24 === * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-15 === * 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison * 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison * 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node === 2024-04-11 === * 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-10 === * 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node === 2024-04-09 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-08 === * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-05 === * 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-02 === * 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk * 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02 === 2024-04-01 === * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node === 2024-03-28 === * 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]]) * 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]]) * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]]) * 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]]) * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]]) * 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3 * 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4 * 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy' * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy' * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy' * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy' === 2024-03-27 === * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2 * 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2 === 2024-03-26 === * 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 * 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) * 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 * 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 * 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2024-03-25 === * 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector * 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector === 2024-03-22 === * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-03-21 === * 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4 * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4 * 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3 * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-19 === * 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-03-18 === * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1 * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-03-16 === * 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8 === 2024-03-15 === * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01 * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) === 2024-03-13 === * 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) * 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) === 2024-03-12 === * 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) * 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) === 2024-03-11 === * 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-03-07 === * 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-05 === * 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-04 === * 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-28 === * 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud * 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-02-22 === * 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-02-21 === * 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-20 === * 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6 * 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6 * 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6 * 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 * 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 * 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 === 2024-02-19 === * 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-02-15 === * 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5 * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 * 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster === 2024-02-13 === * 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4 * 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5 * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5 * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4 * 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4 * 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3 * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3 * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8 * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7 * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7 === 2024-02-12 === * 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-09 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config === 2024-02-08 === * 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6 * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6 * 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6 * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6 * 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10 * 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10 === 2024-02-06 === * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-02-05 === * 09:55 arturo: grant myself member and admin privileges === 2024-01-31 === * 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-29 === * 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-01-26 === * 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster === 2024-01-25 === * 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster === 2024-01-23 === * 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-17 === * 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-12 === * 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1 === 2024-01-11 === * 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-09 === * 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-08 === * 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-05 === * 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-12-26 === * 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable === 2023-12-20 === * 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase === 2023-12-15 === * 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]]) * 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]]) === 2023-12-13 === * 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97) * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec * 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]]) * 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]]) * 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]]) * 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]]) * 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]]) * 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2023-12-12 === * 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) === 2023-12-11 === * 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]]) * 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs * 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]]) * 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]]) === 2023-12-07 === * 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-12-05 === * 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2023-12-04 === * 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-12-01 === * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-11-23 === * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-11-22 === * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-11-20 === * 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-11-17 === * 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-11-09 === * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-11-01 === * 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99) * 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-30 === * 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2023-10-27 === * 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram * 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2023-10-26 === * 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2023-10-25 === * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6 * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6 * 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6 * 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster * 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster === 2023-10-23 === * 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-10-20 === * 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]]) * 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]]) === 2023-10-18 === * 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-10-13 === * 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97) * 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-12 === * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-10 === * 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-09 === * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-05 === * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-04 === * 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2023-10-03 === * 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2023-09-27 === * 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config === 2023-09-25 === * 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2023-09-20 === * 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2023-09-19 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2023-09-15 === * 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-09-14 === * 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer * 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer * 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission * 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission * 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 10:16 dcaro: deploy bulids-api 0.0.96 * 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2023-09-13 === * 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone * 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone === 2023-09-11 === * 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]]) * 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]]) === 2023-09-06 === * 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]]) === 2023-09-05 === * 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) === 2023-08-31 === * 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0) * 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status * 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status * 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0) * 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs * 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0) * 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17 * 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17 * 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17 * 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17 * 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17 * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 === 2023-08-30 === * 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]]) * 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]] === 2023-08-29 === * 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko * 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay === 2023-08-28 === * 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay === 2023-08-22 === * 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko === 2023-08-18 === * 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay * 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay === 2023-08-17 === * 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}} === 2023-08-11 === * 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn === 2023-07-26 === * 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko === 2023-07-25 === * 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko === 2023-07-20 === * 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]]) * 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta === 2023-07-18 === * 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]]) === 2023-07-13 === * 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko === 2023-07-12 === * 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]]) === 2023-07-04 === * 13:55 taavi: removed floating IP and public dns records for the harbor server === 2023-07-03 === * 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko * 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay === 2023-06-26 === * 07:49 dcaro: restarting harbor trove DB (in error status) === 2023-06-21 === * 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]]) * 11:48 dcaro: deploy bulids-api 0.2.0 === 2023-06-16 === * 14:28 dcaro: deployed envvars-api 0.0.1 * 07:41 dcaro: deployed latest builds-api 0.1.0 === 2023-06-15 === * 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye === 2023-06-08 === * 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]]) === 2023-06-07 === * 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko === 2023-06-01 === * 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus * 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus === 2023-05-26 === * 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus * 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus === 2023-05-25 === * 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]]) === 2023-05-24 === * 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]]) * 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]]) === 2023-05-23 === * 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169 === 2023-05-16 === * 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]]) * 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus * 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98 === 2023-05-15 === * 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus * 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus === 2023-05-09 === * 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko * 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2023-05-05 === * 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko === 2023-05-01 === * 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu === 2023-04-30 === * 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko * 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko * 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko === 2023-04-19 === * 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko * 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko * 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko === 2023-04-11 === * 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus * 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo * 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko * 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko * 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko * 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko === 2023-04-05 === * 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko * 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko === 2023-04-03 === * 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo * 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo * 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo * 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo * 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo * 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo * 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo * 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo === 2023-03-19 === * 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko === 2023-03-14 === * 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local * 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local === 2023-03-13 === * 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus === 2023-03-10 === * 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko === 2023-03-09 === * 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko === 2023-03-07 === * 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]] === 2023-03-06 === * 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko * 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko * 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko * 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1 === 2023-03-05 === * 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko === 2023-03-02 === * 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836) === 2023-03-01 === * 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko === 2023-02-28 === * 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko * 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko * 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus === 2023-02-23 === * 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko * 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko === 2023-02-17 === * 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance * 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance === 2023-02-16 === * 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo * 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo * 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko * 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml === 2023-02-15 === * 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo === 2023-02-14 === * 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]] * 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]]) === 2023-02-13 === * 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko * 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo * 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:15 arturo: cordoned & drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]]) * 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo * 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo === 2023-02-10 === * 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo === 2023-02-01 === * 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko === 2023-01-26 === * 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko === 2023-01-23 === * 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko === 2023-01-20 === * 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo === 2023-01-19 === * 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff) === 2023-01-18 === * 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo === 2023-01-17 === * 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance * 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance * 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud === 2023-01-10 === * 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance * 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance === 2022-12-09 === * 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]]) === 2022-11-30 === * 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo === 2022-11-29 === * 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko * 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko * 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko === 2022-11-28 === * 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko * 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko === 2022-11-15 === * 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu === 2022-11-14 === * 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 === 2022-11-07 === * 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169 === 2022-11-04 === * 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo === 2022-11-01 === * 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot === 2022-10-24 === * 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye * 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye * 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus === 2022-10-18 === * 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo === 2022-10-17 === * 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2022-10-14 === * 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko === 2022-10-12 === * 10:29 dcaro: deploying new registry-admission controller === 2022-10-10 === * 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko === 2022-09-28 === * 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]]) === 2022-09-27 === * 14:23 arturo: briefly livehacking puppetmaster === 2022-08-24 === * 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko === 2022-08-12 === * 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again === 2022-08-03 === * 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap * 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station === 2022-08-01 === * 14:01 taavi: unbreak acme-chief after keystone communication issues === 2022-07-19 === * 15:45 taavi: deploying and testing maintain-kubeusers updates === 2022-06-28 === * 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko === 2022-06-24 === * 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko === 2022-06-19 === * 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02 === 2022-06-03 === * 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]]) * 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko * 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko * 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko === 2022-05-30 === * 13:42 taavi: run grid-configurator to remove stale config for some removed nodes === 2022-05-26 === * 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko === 2022-04-20 === * 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko === 2022-04-15 === * 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]] === 2022-04-11 === * 14:47 dcaro: deploying custom version of the regitsry admission hook === 2022-04-08 === * 10:45 arturo: disabled debug mode on the k8s jobs-emailer component === 2022-04-05 === * 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo * 07:21 arturo: deploying toolforge-jobs-framework-cli v7 === 2022-04-04 === * 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo * 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions === 2022-03-25 === * 11:31 dcaro: All alerting VMs rebooted, checking that everything is "working" ([[phab:T304672|T304672]]) * 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]]) * 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]]) * 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]]) === 2022-03-16 === * 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta === 2022-03-15 === * 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo === 2022-03-14 === * 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2022-03-11 === * 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]]) * 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]]) * 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]]) * 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]]) === 2022-03-08 === * 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates === 2022-03-07 === * 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]]) === 2022-02-17 === * 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access === 2022-02-08 === * 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]]) * 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles * 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]]) === 2022-02-04 === * 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]] === 2022-01-28 === * 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo === 2022-01-25 === * 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo === 2022-01-20 === * 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo === 2022-01-19 === * 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta === 2022-01-18 === * 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster * 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 && git cherry-pick FETCH_HEAD * 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix) === 2022-01-17 === * 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo * 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo * 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo === 2022-01-14 === * 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo === 2022-01-12 === * 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo * 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig` * 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig' * 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo === 2022-01-11 === * 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo * 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo === 2021-12-23 === * 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance * 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance * 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance * 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance === 2021-12-22 === * 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance * 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance * 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance * 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance === 2021-12-21 === * 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance * 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]]) === 2021-12-16 === * 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance === 2021-12-15 === * 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance === 2021-12-08 === * 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028 === 2021-11-28 === * 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance === 2021-11-16 === * 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]] === 2021-11-05 === * 19:07 majavah: testing registry-admission changes === 2021-10-28 === * 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true` === 2021-10-25 === * 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]] === 2021-10-20 === * 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories === 2021-10-16 === * 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]] === 2021-10-14 === * 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus * 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus === 2021-10-13 === * 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-12 === * 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master === 2021-10-11 === * 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-07 === * 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-06 === * 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-04 === * 17:07 bstorm: reboot everything [[phab:T291406|T291406]] * 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]] * 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting) * 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]]) === 2021-10-01 === * 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]]) * 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]]) === 2021-09-29 === * 22:13 bstorm: ran label fix script to use new label format * 22:12 bstorm: toollabs-webservice 0.77 deployed === 2021-09-28 === * 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api === 2021-09-27 === * 16:13 majavah: testing volume-admission fix for containers with some volumes mounted === 2021-09-23 === * 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]] === 2021-09-22 === * 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a "fair" test battery against [[phab:T291406|T291406]] === 2021-09-15 === * 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]] === 2021-09-14 === * 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]] * 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]]) * 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]]) * 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]]) === 2021-09-13 === * 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]] === 2021-09-09 === * 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]] * 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]] * 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml * 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp === 2021-09-03 === * 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]] === 2021-08-30 === * 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]] === 2021-08-21 === * 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]] === 2021-08-20 === * 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes === 2021-08-12 === * 16:55 bstorm: deployed updated manifest for ingress-admission * 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]] === 2021-07-30 === * 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]] === 2021-07-29 === * 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]] * 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]] === 2021-07-23 === * 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]] === 2021-07-22 === * 15:32 arturo: re-deploying toolforge-jobs-framework-api === 2021-07-21 === * 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]]) * 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-<nowiki>{</nowiki>apiserver,controller-manager<nowiki>}</nowiki>.yaml in all 3 control nodes ([[phab:T286108|T286108]]) * 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]]) * 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848 === 2021-07-20 === * 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]] * 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]] * 08:36 majavah: resolve merge conflicts on labs/private === 2021-07-16 === * 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]] * 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]]) === 2021-07-15 === * 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}} === 2021-07-14 === * 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]] === 2021-07-12 === * 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues === 2021-07-07 === * 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work === 2021-07-01 === * 17:01 majavah: updating jobs-framework-api * 10:00 arturo: refreshed jobs-api deployment === 2021-06-29 === * 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus * 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus * 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus === 2021-06-28 === * 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus * 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus * 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-25 === * 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-24 === * 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers * 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-22 === * 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18 === 2021-06-17 === * 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again === 2021-06-16 === * 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100 === 2021-06-15 === * 16:10 majavah: set toolsbeta-bastion-05 as grid submit host === 2021-06-14 === * 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]] * 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]]) === 2021-06-12 === * 14:42 majavah: sync hiera key prometheus_nodes to match tools === 2021-06-11 === * 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes * 14:54 majavah: generate and add own root key to passwords::root::extra_keys === 2021-06-08 === * 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]] * 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]] * 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]] * 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work * 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]] === 2021-06-03 === * 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]] * 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]] === 2021-05-25 === * 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 * 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5 * 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020 === 2021-05-24 === * 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes === 2021-05-19 === * 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/ * 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1 === 2021-05-15 === * 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]] === 2021-05-14 === * 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]]) * 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]]) === 2021-05-13 === * 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]] === 2021-05-10 === * 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes * 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]] * 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]] === 2021-05-08 === * 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]] === 2021-05-07 === * 17:00 bstorm: deleted "toolsbeta-test-k8s-haproxy-2", "toolsbeta-test-k8s-haproxy-1" when the dns caches finally dropped [[phab:T282227|T282227]] * 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]] * 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]] * 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/ * 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip` * 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs === 2021-05-05 === * 16:25 Majavah: add self to sudo policy `roots` * 16:07 arturo: grant `taavi` projectadmin (Majavah) === 2021-05-04 === * 10:47 arturo: rebase & resolve merge conflicts in labs/private.git === 2021-05-03 === * 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]]) === 2021-04-29 === * 18:10 bstorm: added and removed an etcd node === 2021-04-23 === * 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was "notready" for some reason === 2021-04-20 === * 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244 === 2021-04-13 === * 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]]) * 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful) * 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]]) * 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]]) === 2021-04-08 === * 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]] === 2021-04-06 === * 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number ([[phab:T267082|T267082]]) === 2021-04-01 === * 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook) * 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs === 2021-03-31 === * 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]] === 2021-03-24 === * 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM * 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]]) * 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01 * 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]]) === 2021-03-23 === * 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]]) * 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]]) === 2021-03-18 === * 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix * 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project) * 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful * 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]]) * 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456 * 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow * 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]]) * 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here * 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]] * 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]] === 2021-03-17 === * 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]]) * 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]]) * 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]]) * 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]]) * 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]]) === 2021-03-16 === * 12:32 arturo: added packages jobutils / misctools v1.41 to <nowiki>{</nowiki>stretch,buster<nowiki>}</nowiki>-toolsbeta aptly repository in tools-sge-services-03 === 2021-03-11 === * 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]] === 2021-03-10 === * 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor === 2021-02-26 === * 20:39 andrewbogott: rebooting all hosts * 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]]) * 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]]) * 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]]) * 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]]) === 2021-02-19 === * 12:42 arturo: deploying new version of the ingress admission controller * 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta * 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]]) * 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]]) === 2021-02-10 === * 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) * 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4 * 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150 (toolsbeta-proxy-3), it was previously pointing to an old IP address === 2021-02-08 === * 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]] === 2021-02-05 === * 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing === 2021-01-21 === * 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]] === 2021-01-13 === * 14:10 dcaro: dcaro doing puppet tests, puppet runs might break * 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]]) * 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]]) === 2020-12-22 === * 10:48 arturo: rebase & resolve ugly git merge conflict in labs/private.git === 2020-12-18 === * 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]]) === 2020-12-14 === * 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]] * 19:25 bstorm: created temporary instance toolsbeta-io-test-local [[phab:T267966|T267966]] === 2020-12-11 === * 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts === 2020-12-10 === * 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]]) === 2020-12-09 === * 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]]) === 2020-12-04 === * 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]]) * 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]]) === 2020-11-23 === * 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]]) * 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]]) * 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]]) * 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]]) * 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]]) * 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]]) * 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]]) === 2020-11-18 === * 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]]) * 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]]) === 2020-11-17 === * 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]]) * 12:09 Lucas_WMDE: <dcaro> 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]]) * 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]]) * 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]]) * 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]]) === 2020-11-16 === * 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]]) * 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]]) === 2020-11-10 === * 19:42 bstorm: safelisted "argocd" namespace with namespaceSelector for registry-admission controller * 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it * 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]]) * 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]]) * 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]]) * 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]]) === 2020-11-06 === * 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]] * 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]] * 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]]) * 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]]) * 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]]) * 13:18 dcaro: bringin up a new proxy-4 instance as slave === 2020-11-05 === * 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]]) * 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]]) === 2020-11-04 === * 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]]) * 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03 * 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use * 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use * 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix === 2020-11-03 === * 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]]) * 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing * 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04 * 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02 === 2020-11-02 === * 13:35 arturo: added dcaro as projectadmin & user ([[phab:T266068|T266068]]) === 2020-10-29 === * 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]]) === 2020-10-28 === * 18:58 andrewbogott: deleting toolsbeta-puppetmaster-03 — seems broken and unused === 2020-10-22 === * 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]] === 2020-09-10 === * 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive) * 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]]) * 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]]) * 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]]) === 2020-09-09 === * 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic. * 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down * 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least * 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them * 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]]) * 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]]) * 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]]) === 2020-07-15 === * 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]] === 2020-07-14 === * 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing === 2020-07-08 === * 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]]) === 2020-06-26 === * 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]]) === 2020-06-24 === * 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]]) * 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]]) * 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]]) * 11:26 arturo: add TXT record `"v=spf1 mx -all"` [[phab:T120225|T120225]] * 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]] === 2020-06-23 === * 13:10 arturo: added herron to the test tool for email testing * 11:36 arturo: removing `benapetr` and adding myself to the test tool * 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix * 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group * 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org * 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert * 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251 === 2020-06-16 === * 22:54 bd808: Building webservice 0.72 === 2020-06-15 === * 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]] * 17:52 bd808: Building webservice 0.71 === 2020-06-12 === * 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]] === 2020-06-11 === * 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there * 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working * 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O<nowiki>{</nowiki>project:toolsbeta<nowiki>}</nowiki>' 'run-puppet-agent'` we are mostly back in business * 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04 * 12:14 arturo: poweroff toolsbeta-puppetmaster-03 * 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04 * 11:53 arturo: create VM toolsbeta-puppetmaster-04 * 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults * 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems * 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03 * 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03 * 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` === 2020-06-04 === * 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap === 2020-05-28 === * 11:27 arturo: cleanup livehackings * 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]]) * 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 === 2020-05-27 === * 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]]) * 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]]) * 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]]) * 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]]) * 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]]) === 2020-05-26 === * 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]] * 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]]) * 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]]) * 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]]) * 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]]) * 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]]) * 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]] * 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again === 2020-05-21 === * 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py === 2020-05-19 === * 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]]) * 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb` === 2020-05-15 === * 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]] * 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts === 2020-05-14 === * 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]]) * 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]]) === 2020-05-12 === * 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]] * 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]] * 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]] === 2020-05-08 === * 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]]) * 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]]) === 2020-05-07 === * 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]]) * 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03 * 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]]) === 2020-05-06 === * 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta * 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]]) === 2020-05-05 === * 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]]) * 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]]) === 2020-05-04 === * 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project === 2020-04-29 === * 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]] === 2020-04-20 === * 14:47 arturo: added joakino to toolsbeta.admin LDAP group * 12:06 arturo: installing tools-webservice v0.68 for testing * 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there * 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project === 2020-04-10 === * 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]] * 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]] * 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up === 2020-04-08 === * 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]] === 2020-03-31 === * 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta === 2020-03-30 === * 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point) * 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use === 2020-03-24 === * 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]] === 2020-03-19 === * 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]] * 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]]) === 2020-03-16 === * 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]] * 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]] * 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]] * 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]] * 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well * 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]]) * 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]]) * 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests * 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]]) === 2020-03-11 === * 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta === 2020-03-09 === * 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]]) * 13:08 arturo: instance quota was full, bump it from 35 to 40 === 2020-03-06 === * 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names === 2020-03-05 === * 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test === 2020-02-27 === * 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing * 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3] * 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers === 2020-02-26 === * 20:35 bstorm_: hard rebooting the grid master for toolsbeta * 20:20 jeh: restart toolsbeta-sgegrid-shadow === 2020-02-18 === * 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists === 2020-02-10 === * 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]] === 2020-02-07 === * 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]] * 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]] === 2020-01-23 === * 03:14 bd808: Demoted projectadmins not listed in the "roots" sudoer policy to project members just to avoid random confusion * 03:06 bd808: Added legoktm to "roots" sudoer policy * 02:53 bd808: Added legoktm as project admin === 2020-01-22 === * 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-<nowiki>{</nowiki>1,2<nowiki>}</nowiki>, source of cronspam === 2020-01-21 === * 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1 * 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]] === 2020-01-17 === * 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]]) * 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me) === 2020-01-16 === * 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out * 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]]) === 2020-01-14 === * 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch === 2020-01-13 === * 16:41 bstorm_: There was a filesystem unclean and other problems on the "old cluster" worker node 1001. Rebooting it in case that helps. === 2020-01-10 === * 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing === 2020-01-07 === * 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]] === 2020-01-06 === * 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things === 2020-01-03 === * 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged * 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]]) === 2020-01-02 === * 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly. * 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups === 2019-12-17 === * 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes === 2019-12-16 === * 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file === 2019-12-04 === * 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use * 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use === 2019-12-02 === * 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster * 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]]) === 2019-11-25 === * 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]]) === 2019-11-21 === * 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet) === 2019-11-15 === * 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]] === 2019-11-14 === * 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug === 2019-11-08 === * 19:36 bstorm_: rebooted the proxy server just in case that fixes something. * 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]]) * 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet === 2019-11-07 === * 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]] === 2019-11-06 === * 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]] * 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]] * 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]] * 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]] * 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized) * 17:35 bstorm_: changing some hiera to work with new proxy host * 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]]) === 2019-11-05 === * 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]] === 2019-10-25 === * 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]] * 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2 === 2019-10-23 === * 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]] * 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]]) * 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]]) * 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]]) === 2019-10-22 === * 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]] * 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]] * 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]] * 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}} * 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]] * 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}} * 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]] === 2019-10-21 === * 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}} * 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]] === 2019-10-18 === * 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud` * 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused) * {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}} * {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}} * 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused) === 2019-10-14 === * 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required * 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]] === 2019-10-09 === * 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]]) === 2019-10-08 === * 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]] * 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]] === 2019-10-07 === * 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue === 2019-09-25 === * 23:31 bd808: Updated user list for "roots" sudoer policy * 23:30 bd808: Granted Krenair projectadmin === 2019-09-05 === * {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}} === 2019-08-12 === * 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]] === 2019-08-09 === * 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003 === 2019-07-24 === * 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]] * 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster === 2019-07-18 === * 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-* * 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]]) * 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code * {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}} === 2019-07-17 === * 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]] * 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]] === 2019-07-15 === * 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix * 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]] === 2019-07-03 === * 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]]) * 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]]) * 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]] === 2019-07-02 === * 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]]) * 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap === 2019-07-01 === * 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]] * 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]] * 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]] * 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]] * 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix === 2019-06-28 === * 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm === 2019-06-25 === * 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]] * 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]] === 2019-06-21 === * 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]] === 2019-06-19 === * 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]]) === 2019-06-14 === * 16:24 bstorm_: Manually failed "back" to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta * 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything * 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on. * 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now === 2019-06-06 === * 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]` * 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix * 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix === 2019-06-05 === * 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets) * 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]] * 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]] * 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment) * 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused === 2019-05-24 === * 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon * 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs * 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01 * 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01: * 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc === 2019-05-07 === * 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix * 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix * 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix === 2019-04-25 === * 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver: toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master === 2019-04-12 === * 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1 * 00:05 andrewbogott: migrating remaining VMs to eqiad1-r === 2019-03-25 === * 18:00 bd808: All Trusty instances shutdown and now in process of deleting * 17:42 bd808: Preparing to shutdown beta Trusty job grid === 2019-03-22 === * 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]] === 2019-03-15 === * 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]]) === 2019-02-20 === * 14:58 andrewbogott: moving toolsbeta-grid-master and toolsbeta-puppetmaster-02 to labvirt1003 === 2019-02-14 === * 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002 === 2018-12-04 === * 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/ === 2018-11-26 === * 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03 * 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM === 2018-11-08 === * 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]]) === 2018-10-22 === * 21:20 bstorm_: launched a stretch/sonofgridengine master server === 2018-09-19 === * 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]] * 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]]) * 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb === 2018-07-22 === * 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]] === 2018-07-18 === * 10:46 harej: Deleted toolsbeta-flynn-01 === 2018-07-12 === * 23:06 bstorm_: Got the grid master running === 2018-06-28 === * 16:34 chicocvenancio: adding harej as root for flynn testing === 2018-06-27 === * 22:35 chicocvenancio: add harej as project admin to test Flynn stuff === 2018-06-22 === * 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning * 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions * 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating * 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01 * 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice * 07:56 andrewbogott: someone removed /usr/bin/webservice === 2018-05-15 === * 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]] * 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01 === 2018-05-08 === * 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]] === 2018-05-07 === * 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs * 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01 * 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2> /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)` * 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd === 2018-05-06 === * 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd === 2018-05-05 === * 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]] * 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml` === 2018-05-02 === * 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera * 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]] * 20:31 zhuyifei1999_: nuke webservice instances and rebuild * 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]] === 2018-04-20 === * 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem === 2018-04-19 === * 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :( * 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead * 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org => toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs * 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]] === 2018-03-30 === * 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]] === 2018-03-14 === * 18:07 chicocvenancio: updated paws-beta k8s cluster and nodes to v1.9.4 for [[phab:T189680|T189680]] === 2018-03-05 === * 19:33 chicocvenancio: added Zhuyifei1999 as project admin === 2018-02-09 === * 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]]) === 2017-08-07 === * 14:09 andrewbogott: deleted etcd-k8s-CTEST and k8s-master-CTEST === 2017-04-26 === * 15:38 madhuvishy: add Madhuvishy as projectadmin === 2016-10-07 === * 19:30 valhallasw`cloud: (puppet certs, to be precise) * 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs === 2016-10-04 === * 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for) === 2016-09-08 === * 17:11 bd808: Added BryanDavis (self) to project as admin === 2016-08-29 === * 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck * 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck * 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains === 2016-07-03 === * 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010 === 2016-05-27 === * 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs === 2016-05-26 === * 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host === 2015-10-13 === * 19:21 valhallasw`cloud: started building toolsbeta-bastion. === 2015-09-07 === * 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest... * 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages) === July 4 === * 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22 === July 2 === * 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting === July 1 === * 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it? * 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login === June 7 === * 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo * 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config === May 11 === * 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages === May 2 === * 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code * 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera * 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in * 10:25 valhallasw`cloud: set Hiera variable "elasticsearch::cluster_name": toolsbeta-logstash-eqiad * 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]]) === April 26 === * 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now * 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test === March 31 === * 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources. It can be restarted when needed. === September 20 === * 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006 === July 22 === * 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-) === August 19 === * 12:44 petan: rebooting apache it seems to be frozen === August 4 === * 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-) === July 6 === * 19:42 petan: rebooting login === June 26 === * 08:03 wm-bot: petrb: updating logsplitter === June 24 === * 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info * 13:43 scfc_de: Made scfc root. * 13:42 scfc_de: Created toolsbeta-puppetmaster. * 11:09 YuviPanda: Granted yuvipanda root on toolsbeta === June 21 === * 13:46 wm-bot: petrb: rebooting all servers === June 17 === * 08:31 petan: switching all instances to nfs === June 16 === * 15:37 petan: importing sudo policies of tools * 15:36 petan: importing security groups of tools * 15:36 petan: blah {{SAL|Project Name=toolsbeta}} <noinclude>[[Category:SAL]]</noinclude> ok4f8e10i3ui1cukjjhrvjyj7zxf03i 0 7919 2466268 2466265 2026-10-10T12:00:09Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2466268 wikitext text/x-wiki == 2026-10-10 == * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-09 == * 18:54 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) * 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment * 18:48 cscott@deploy1003: cscott, ssastry: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:44 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] * 17:38 Daimona: Run database query from [[phab:T440557|T440557]]#12411029 in x1.wikishared * 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 * 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet * 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606{{!}}T440606]]' * 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet * 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet * 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet * 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet * 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location * 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet * 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet * 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet * 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet * 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test * 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test * 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test * 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm * 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A * 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch * 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump * 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm * 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet * 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet * 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet * 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) * 01:25 kemayo@deploy1003: kemayo: Continuing with deployment * 01:24 kemayo@deploy1003: kemayo: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 01:20 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] * 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr [[phab:T440068|T440068]] - downloads 120x times faster == 2026-10-08 == * 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning * 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) * 23:16 ejegg@deploy1003: ejegg: Continuing with deployment * 23:11 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:51 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] * 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) * 22:25 kemayo@deploy1003: kemayo: Continuing with deployment * 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 22:20 kemayo@deploy1003: kemayo: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 22:00 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] * 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) * 21:37 ejegg@deploy1003: ejegg: Rolling back deployment * 21:27 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad * 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:07 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] * 20:48 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) * 20:43 cjming@deploy1003: cjming: Continuing with deployment * 20:42 cjming@deploy1003: cjming: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:38 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] * 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] (duration: 09m 41s) * 20:27 mlitn@deploy1003: mlitn: Continuing with deployment * 20:27 mlitn@deploy1003: mlitn: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:23 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] * 20:21 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) * 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment * 20:10 cjming@deploy1003: milazg, cjming: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] * 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) * 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update * 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet * 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet * 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet * 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes * 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts * 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s) * 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad * 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change * 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie * 16:07 swfrench@dns1004: END - running authdns-update * 16:04 swfrench@dns1004: START - running authdns-update * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) * 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet * 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet * 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet * 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning * 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet * 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet * 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes * 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:54 cmooney@dns3003: END - running authdns-update * 14:51 cmooney@dns3003: START - running authdns-update * 14:50 cgoubert@dns2004: END - running authdns-update * 14:48 cgoubert@dns2004: START - running authdns-update * 14:48 moritzm: installing pyasn1 security updates * 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet * 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] (duration: 36m 09s) * 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet * 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 14:23 moritzm: installing perl security updates * 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet * 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 14:03 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:47 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) * 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet * 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment * 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet * 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet * 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet * 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet * 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet * 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # [[phab:T440023|T440023]] [[phab:T439902|T439902]] * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet * 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet * 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet * 12:30 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) * 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet * 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet * 12:25 zabe@deploy1003: zabe: Continuing with deployment * 12:24 zabe@deploy1003: zabe: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet * 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] * 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet * 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet * 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet * 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet * 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet * 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] (duration: 04m 15s) * 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet * 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet * 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet * 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] * 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:10 topranks: cr1-codfw switch back to master routing engine [[phab:T426197|T426197]] * 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet * 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:36 topranks: cr1-codfw switch to backup routing engine [[phab:T426197|T426197]] * 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host * 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet * 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet * 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet * 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover * 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet * 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet * 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host * 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade [[phab:T417871|T417871]] * 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet * 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet * 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet * 08:59 moritzm: installing Wireshark security updates * 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json * 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json * 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - [[phab:T436922|T436922]] * 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json * 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 [[phab:T436922|T436922]] * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet * 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet * 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:42 moritzm: installing python3.13 security updates * 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet * 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:01 moritzm: installing mbedtls security updates * 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet * 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet * 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) * 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet * 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment * 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet * 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet * 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet * 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet * 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups * 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet * 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet * 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet * 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet * 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old ([[phab:T440358|T440358]]) * 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet * 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet * 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet * 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet * 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet * 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet * 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet * 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet * 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet * 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet * 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet * 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet * 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet * 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet * 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet * 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet * 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet * 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet * 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet * 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet * 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet * 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet * 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet * 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet * 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet * 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet * 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet * 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet * 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet * 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet * 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet * 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet * 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font ([[phab:T393763|T393763]]) == 2026-10-07 == * 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm * 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet * 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) * 20:23 kemayo@deploy1003: kemayo: Continuing with deployment * 20:21 kemayo@deploy1003: kemayo: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] * 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply * 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply * 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts * 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s) * 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet * 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] (duration: 10m 40s) * 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] * 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet * 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet * 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for [[phab:T417800|T417800]] (duration: 05m 23s) * 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:17 blake@deploy1003: Started scap sync-world: add restart policies for [[phab:T417800|T417800]] * 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet * 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet * 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) * 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet * 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) * 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet * 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) * 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade * 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade * 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet * 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting * 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade * 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade * 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade * 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade * 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade * 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) * 14:48 jforrester@deploy1003: jforrester: Continuing with deployment * 14:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams * 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams * 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs * 14:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] * 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs * 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin * 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo * 14:34 jforrester@deploy1003: jforrester: Continuing with deployment * 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo * 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:27 moritzm: installing PHP 8.2 security updates * 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru * 14:26 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet * 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet * 14:22 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] * 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 14:19 moritzm: installing postgresql-15 security updates * 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - [[phab:T439840|T439840]] * 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet * 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet * 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment * 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:54 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet * 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 13:44 papaul: start maintenance on cr2-codfw * 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) * 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet * 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] * 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) * 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:13 moritzm: installing mbedtls security updates * 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) * 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet * 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply * 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet * 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet * 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet * 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet * 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet * 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet * 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet * 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet * 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 andre: Rolling back again wmf.23 from group1 to group0 - [[phab:T438219|T438219]] * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet * 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet * 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for [[phab:T417800|T417800]] (duration: 07m 32s) * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for [[phab:T417800|T417800]] * 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet * 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet * 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet * 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet * 09:54 duesen: just a test for SAL * 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:50 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) * 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet * 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:36 daniel@deploy1003: daniel: Continuing with deployment * 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet * 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet * 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:26 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet * 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet * 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet * 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet * 09:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:13 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] * 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) * 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] * 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) * 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] * 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet * 08:20 aklapper@deploy1003: Rolling back deployment * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet * 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet * 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet * 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet * 08:02 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) * 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment * 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet * 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia [[phab:T392465|T392465]] * 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:25 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] * 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet * 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet * 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet * 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 moritzm: installing bird security updates * 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet * 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet * 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet * 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet * 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet * 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet * 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet * 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:10 kart_: Updated Apertium to 2026-09-30-051826-production ([[phab:T437213|T437213]]) * 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply * 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply * 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply * 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply * 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host [[phab:T437279|T437279]] * 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning * 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host * 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply * 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-06 == * 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) * 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:00 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] * 20:15 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 * 20:10 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt * 20:03 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] * 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test * 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet * 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet * 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test * 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet * 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet * 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test * 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet * 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet * 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet * 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet * 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet * 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet * 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet * 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet * 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet * 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:39 bking@dns2004: END - running authdns-update * 16:37 bking@dns2004: START - running authdns-update * 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet * 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet * 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet * 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet * 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:26 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) * 15:21 daniel@deploy1003: daniel: Continuing with deployment * 15:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing python3.13 security updates * 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 15:11 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] * 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw * 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] (duration: 00m 42s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] * 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment * 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] (duration: 00m 51s) * 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] * 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment * 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:57 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet * 14:48 moritzm: restarting pybal on lvs1020 [[phab:T439730|T439730]] * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw * 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet * 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) * 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment * 14:33 moritzm: restarting pybal on lvs2012 [[phab:T439730|T439730]] * 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet * 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet * 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) * 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet * 14:23 moritzm: restarting pybal on lvs2014 [[phab:T439730|T439730]] * 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb * 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb * 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet * 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet * 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb * 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet * 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet * 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding chec}} * 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment * 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin * 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet * {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a s}} * 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet * 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding check}} * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment * 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] * 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 moritzm: installing php8.2 security updates * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 214m 10s) * 13:31 slyngshede@dns1004: END - running authdns-update * 13:29 slyngshede@dns1004: START - running authdns-update * 13:25 moritzm: installing libssh2 security updates * 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 13:14 moritzm: installing glibc security updates * 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 11:33 moritzm: installing chromium security updates * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply * 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply * 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 * 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back * 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor ([[phab:T406724|T406724]]) * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading * 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 [[phab:T436082|T436082]] * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 moritzm: installing curl security updates * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing * 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m {{!}} tee gc_run_06102026` - [[phab:T437297|T437297]] * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 * 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 00m 19s) * 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 * 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back * 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 * 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing * 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing * 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning * 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning * 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning * 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 * 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back * 08:00 matthiasmullie: UTC morning backport window done * 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) * 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:50 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] * 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) * 07:38 dcausse@deploy1003: dcausse: Continuing with deployment * 07:38 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 07:31 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] * 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 * 07:21 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) * 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back * 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment * 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] * 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 * 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 * 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning * 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 * 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) * 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] (duration: 27m 17s) * 03:28 mwpresync@deploy1003: Rolling back deployment * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-05 == * 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) * 23:18 derenrich@deploy1003: derenrich: Continuing with deployment * 23:16 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:12 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet * 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439719|T439719]] * 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439974|T439974]] * 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:51 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439932|T439932]] * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams * 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs * 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs * 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb) * 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # [[phab:T439577|T439577]] * 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) * 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes * 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply * 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply * 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru * 16:32 dcausse: closing the "Distributed search queries experiment" window * 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) * 16:25 dcausse@deploy1003: dcausse: Continuing with deployment * 16:23 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox * 16:19 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] * 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad * 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) * 15:59 dcausse@deploy1003: dcausse: Continuing with deployment * 15:58 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:54 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] * 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad * 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] (duration: 25m 12s) * 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm * 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet * 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox * 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:27 dcausse@deploy1003: dcausse: Continuing with deployment * 15:26 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet * 15:22 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm * 15:16 elukey@dns1004: END - running authdns-update * 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] (duration: 10m 11s) * 15:14 elukey@dns1004: START - running authdns-update * 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet * 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:10 dcausse@deploy1003: dcausse: Continuing with deployment * 15:09 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:05 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet * 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] (duration: 10m 12s) * 14:53 dcausse@deploy1003: dcausse: Continuing with deployment * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:52 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet * 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) * 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] (duration: 10m 04s) * 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet * 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet * 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 dcausse@deploy1003: dcausse: Continuing with deployment * 14:32 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:28 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] * 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) * 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:15 dcausse@deploy1003: dcausse: Continuing with deployment * 14:13 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:09 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439968|T439968]] * 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet * 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet * 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] (duration: 09m 59s) * 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service * 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:37 zabe@deploy1003: zabe: Continuing with deployment * 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:36 zabe@deploy1003: zabe: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] * 13:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) * 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet * 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet * 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet * 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:21 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] * 13:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet * 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet * 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment * 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet * 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet * 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet * 13:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet * 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet * 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet * 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie * 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) * 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:31 moritzm: installing sqlite3 security updates * 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment * 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:16 moritzm: installing mesa security updates * 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org * 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org * 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org * 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org * 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005 * 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm * 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet * 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] * 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet * 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet * 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet * 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet * 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet * 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:46 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] (duration: 04m 44s) * 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:42 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] * 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing * 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests * 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing * 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests * 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm * 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests * 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests * 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing * 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing * 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet * 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet * 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing * 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm * 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests * 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests * 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet * 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet * 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie * 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for [[phab:T417800|T417800]] (duration: 09m 40s) * 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for [[phab:T417800|T417800]] * 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet * 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet * 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:22 volans: upgrading cloudcumin2001 in-place to trixie [[phab:T436667|T436667]] * 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm * 07:49 dpogorzelski@dns1004: END - running authdns-update * 07:46 dpogorzelski@dns1004: START - running authdns-update * 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-04 == * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-03 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-02 == * 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` [[phab:T440033|T440033]] * 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - [[phab:T427353|T427353]] * 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie * 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet * 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet * 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet * 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet * 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet * 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet * 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet * 17:44 dzahn@dns1004: END - running authdns-update * 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet * 17:41 dzahn@dns1004: START - running authdns-update * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet * 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet * 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 * 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 * 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox * 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 * 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet * 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet * 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes * 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) * 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes * 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet * 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm * 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet * 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox * 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm * 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox * 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts * 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm * 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox * 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts * 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-10-01 == * 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.* * 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) * 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] * 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — [[phab:T432444|T432444]] * 21:24 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) * 21:19 egardner@deploy1003: egardner: Continuing with deployment * 21:17 egardner@deploy1003: egardner: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] * 21:06 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) * 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment * 20:59 brennen@deploy1003: brennen, srapoj: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] * 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - [[phab:T432444|T432444]] * 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) * 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:37 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] * 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) * 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment * 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:18 cjming@deploy1003: cjming, volker-e: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can * 20:14 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] * 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two * 19:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] (duration: 09m 40s) * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:46 zabe@deploy1003: zabe: Continuing with deployment * 19:46 zabe@deploy1003: zabe: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:42 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] * 19:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) * 19:36 zabe@deploy1003: zabe: Continuing with deployment * 19:36 zabe@deploy1003: zabe: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] * 19:19 jhathaway: puppet re-enabled * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 * 18:25 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): logs kind of a mess, but no current blockers, rolling to all wikis * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox * 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T439100|T439100]] (duration: 33m 25s) * 17:16 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T439100|T439100]] * 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:32 moritzm: installing socat security updates * 15:29 dpogorzelski@dns1004: END - running authdns-update * 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:25 dpogorzelski@dns1004: START - running authdns-update * 15:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] (duration: 10m 12s) * 15:14 zabe@deploy1003: zabe: Continuing with deployment * 15:14 zabe@deploy1003: zabe: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing postgresql security updates * 15:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] * 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.* * 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.* * 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release * 14:48 moritzm: installing busybox bugfix updates from trixie point release * 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) * 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment * 14:42 moritzm: installing zsh bugfix update from trixie point release * 14:41 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] * 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release * 14:31 dpogorzelski@dns1004: END - running authdns-update * 14:29 dpogorzelski@dns1004: START - running authdns-update * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 14:04 awight: EU afternoon backport complete * 14:03 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) * 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:58 awight@deploy1003: lilients, awight: Continuing with deployment * 13:57 awight@deploy1003: lilients, awight: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:53 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:41 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) * 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:36 awight@deploy1003: awight, migr: Continuing with deployment * 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 awight@deploy1003: awight, migr: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve * 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:18 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] * 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment * 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:34 moritzm: installing libtasn1-6 security updates * 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) * 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:22 moritzm: installing glibc security updates * 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] * 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) * 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] * 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad * 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) * 11:50 jforrester@deploy1003: jforrester: Continuing with deployment * 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad * 11:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] * 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw * 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw * 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) * 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment * 10:44 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # [[phab:T439805|T439805]] * 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) * 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 * 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates * 08:05 moritzm: installing pcre2/openssl security updates * 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o) * 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) * 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:16 kharlan@deploy1003: kharlan: Continuing with deployment * 07:15 kharlan@deploy1003: kharlan: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] * 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:54 kharlan@deploy1003: kharlan: Continuing with deployment * 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:53 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet * 06:33 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] * 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) * 06:23 kharlan@deploy1003: kharlan: Continuing with deployment * 06:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:18 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] == 2026-09-30 == * 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 21:22 sbassett: Deployed updated mitigation for [[phab:T435455|T435455]] * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:52 arlolra@deploy1003: arlolra: Continuing with deployment * 20:52 arlolra@deploy1003: arlolra: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:47 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] * 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment * 20:31 mstyles@deploy1003: mstyles, catrope: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] * 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage * 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod * 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:22 brennen@deploy1003: Rolling back deployment * 18:10 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): no current blockers, rolling to group1 * 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:15 bking@dns2004: END - running authdns-update * 17:13 bking@dns2004: START - running authdns-update * 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet * 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet * 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet * 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:54 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet * 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:49 zabe@deploy1003: zabe: Continuing with deployment * 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:48 zabe@deploy1003: zabe: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet * 14:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] * 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet * 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet * 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet * 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) * 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:30 jforrester@deploy1003: jforrester: Continuing with deployment * 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:29 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) * 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: jforrester: Continuing with deployment * 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:16 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] * 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] (duration: 11m 33s) * 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 Amir1: mwscript-k8s --comment="[[phab:T358266|T358266]]" --local_dblist=<(expanddblist all {{!}} grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing ([[phab:T358266|T358266]]) * 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary [[phab:T437937|T437937]] * 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment * 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] * 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) * 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment * 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet * 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet * 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet * 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment * 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:25 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet * 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] * 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet * 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet * 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment * 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:10 samtar@deploy1003: migr, samtar: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] * 13:05 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] (duration: 10m 41s) * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet * 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet * 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) * 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - [[phab:T439667|T439667]] * 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet * 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:26 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] * 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet * 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet * 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet * 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 11:07 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] (duration: 11m 12s) * 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment * 10:59 cgoubert@deploy1003: cgoubert: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:56 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet * 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - [[phab:T438454|T438454]] * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:08 btullis@dns1004: END - running authdns-update * 09:06 btullis@dns1004: START - running authdns-update * 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' * 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' * 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad * 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad * 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca * 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw * 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw * 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:46 elukey@dns1004: END - running authdns-update * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:44 elukey@dns1004: START - running authdns-update * 07:35 moritzm: installing pcre2/openssl security updates * 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] (duration: 11m 27s) * 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:09 dcausse@deploy1003: dcausse: Continuing with deployment * 07:07 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] * 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet * 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet * 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-29 == * 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 * 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 * 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 * 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox * 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 * 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 * 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - [[phab:T427353|T427353]] [[phab:T395938|T395938]] * 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) * 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment * 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] * 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) * 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 * 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 * 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:43 cklimas@deploy1003: cklimas: Continuing with deployment * 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:38 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:33 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] * 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox * 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 * 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) * 20:19 sbassett@deploy1003: sbassett: Rolling back deployment * 20:12 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm * 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:08 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet * 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 18:06 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]) - no current blockers, rolling to group0 * 18:05 sandraebele: Deployed Refinery as part of weekly deployment train * 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) * 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] * 17:53 sandraebele: Deploying Refinery as part of weekly deployment train * 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) * 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] * 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie * 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train * 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 * 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` [[phab:T436186|T436186]] * 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # [[phab:T418194|T418194]] * 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply * 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply * 15:59 jhathaway: updating puppet stdlib, should be a noop * 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 14:21 moritzm: installing rsyslog security updates * 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet * 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet * 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw * 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:05 krinkle@deploy1003: krinkle: Continuing with deployment * 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox * 14:00 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 13:56 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] * 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] (duration: 00m 09s) * 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) * 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] * 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:25 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) * 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:20 samtar@deploy1003: samtar: Continuing with deployment * 13:19 samtar@deploy1003: samtar: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:14 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] * 13:10 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) * 13:05 samtar@deploy1003: samtar: Continuing with deployment * 13:04 samtar@deploy1003: samtar: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:00 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] * 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging * 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) * 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment * 12:44 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] * 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — [[phab:T432444|T432444]] * 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 * 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for [[phab:T417800|T417800]] (duration: 04m 14s) * 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for [[phab:T417800|T417800]] * 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox * 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet * 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # [[phab:T418194|T418194]] * 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) * 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment * 08:52 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] * 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad * 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad * 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad * 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:10 moritzm: installing PHP 8.4 security updates * 07:49 matthiasmullie: UTC morning backport window done * 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) * 07:44 mlitn@deploy1003: mlitn: Continuing with deployment * 07:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:39 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] * 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) * 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 07:30 kharlan@deploy1003: kharlan: Continuing with deployment * 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 07:27 kharlan@deploy1003: kharlan: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:26 kart_: Updated cxserver to 2026-09-25-114231-production * 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 07:23 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] * 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org * 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) * 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org * 07:11 mlitn@deploy1003: mlitn: Continuing with deployment * 07:09 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now * 07:04 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] * 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply * 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply * 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm * 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm * 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm * 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm * 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm * 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure * 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 * 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033 * 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] (duration: 36m 03s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie == 2026-09-28 == * 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:35 zabe: run script to fix fr_metadata drifts # [[phab:T420341|T420341]] * 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 21:47 sbassett: Deployed security mitigation for [[phab:T435455|T435455]] * 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync * 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync * 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - [[phab:T439201|T439201]] * 21:13 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) * 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment * 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 cjming@deploy1003: volker-e, cjming: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:00 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] * 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) * 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia [[phab:T438375|T438375]] * 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 20:46 kemayo@deploy1003: kemayo: Continuing with deployment * 20:45 kemayo@deploy1003: kemayo: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] * 20:32 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) * 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie * 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment * 20:26 cjming@deploy1003: ssastry, cjming: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:22 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] * 20:16 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) * 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment * 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.* * 20:07 aude@deploy1003: aude, volker-e: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:03 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie * 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts * 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s) * 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... * 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts * 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s) * 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie * 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie * 16:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) * 16:15 zabe@deploy1003: zabe: Continuing with deployment * 16:10 zabe@deploy1003: zabe: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:06 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] * 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] (duration: 19m 54s) * 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # [[phab:T439319|T439319]] * 15:45 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw * 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw * 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 14:55 Lucas_WMDE: UTC afternoon backport+config window done * 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) * 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment * 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) * 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] * 13:58 vgutierrez: repool cp7001 - [[phab:T439361|T439361]] * 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) * 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment * 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - [[phab:T432444|T432444]] * 13:47 bking@dns2004: END - running authdns-update * 13:44 bking@dns2004: START - running authdns-update * 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] * 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw * 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) * 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 vgutierrez: depool cp7001 for some purged tests - [[phab:T439361|T439361]] * 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw * 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw * 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] * 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) * 13:11 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] * 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet * 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:28 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) * 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:22 zabe@deploy1003: zabe: Continuing with deployment * 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet * 11:21 zabe@deploy1003: zabe: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] * 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 10:25 moritzm: installing chromium security updates * 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet * 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet * 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor * 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:16 moritzm: installing Exim security updates * 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:59 matthiasmullie: UTC morning backport window complete * 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) * 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment * 07:49 mlitn@deploy1003: migr, mlitn: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:44 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] * 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] (duration: 32m 06s) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:27 mlitn@deploy1003: mlitn: Continuing with deployment * 07:26 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi * 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] * 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-27 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-26 == * 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) * 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 * 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] (duration: 10m 53s) * 08:02 oblivian@deploy1003: oblivian: Continuing with deployment * 08:00 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:56 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] * 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] (duration: 11m 01s) * 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 03:30 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-25 == * 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values (duration: 00m 11s) * 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values * 17:59 cdanis@dns1004: END - running authdns-update * 17:57 cdanis@dns1004: START - running authdns-update * 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie * 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet * 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - [[phab:T343000|T343000]] * 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* * 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # [[phab:T438750|T438750]] * 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet * 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active * 13:24 Emperor: repool sessionstore in codfw * 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet * 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox * 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet * 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet * 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # [[phab:T438366|T438366]] * 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet * 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - [[phab:T439105|T439105]] * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet * 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet * 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox * 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet * 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.* * 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.* * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet == 2026-09-24 == * 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate * {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on t}} * 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment * 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment}} * {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on te}} * 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.* * 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet * 21:45 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s) * 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:40 catrope@deploy1003: catrope: Continuing with deployment * 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet * 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet * 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:33 catrope@deploy1003: catrope: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:28 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] * 21:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] (duration: 19m 27s) * 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment * 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi * 21:08 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] * 21:04 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s) * 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.* * 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.* * 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment * 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:53 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] * 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:49 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s) * 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:44 catrope@deploy1003: catrope: Continuing with deployment * 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:43 catrope@deploy1003: catrope: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet * 20:39 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s) * 20:14 kemayo@deploy1003: kemayo: Continuing with deployment * 20:12 kemayo@deploy1003: kemayo: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 20:08 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] * 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance * 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro * 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet` * 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet * 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness * 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance * 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly * 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance * 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw * 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance * 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters * 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian{{!}}tegola-vector-tiles) * 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s) * 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment * 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] * 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance * 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance * 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance * 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance * 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw * 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance * 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance * 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance * 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance * 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability * 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s) * 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.* * 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw * 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.* * 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift * 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master{{!}}device-analytics{{!}}echostore{{!}}helm-charts{{!}}k8s-ingress-wikikube-ro{{!}}linkrecommendation{{!}}mathoid{{!}}restbase{{!}}restbase-async{{!}}rest-gateway-ro{{!}}mobileapps{{!}}mwdebug.*{{!}}push-notifications{{!}}recommendation-api{{!}}releases{{!}}wikifeeds * 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid{{!}}zotero * 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium{{!}}schema{{!}}termbox{{!}}proton{{!}}cxserver * 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor * 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.* * 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.* * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro * 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions * 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s) * 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo * 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet * 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie * 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.* * 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes * 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet * 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] * 15:35 vgutierrez@dns1004: END - running authdns-update * 15:33 vgutierrez@dns1004: START - running authdns-update * 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s) * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet * 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie * 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox * 15:06 vgutierrez@dns1004: END - running authdns-update * 15:04 vgutierrez@dns1004: START - running authdns-update * 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update * 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s) * 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.* * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.* * 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:51 dkertesz: repooling cp7001{{!}}7011 after successful testing ([[phab:T343000|T343000]]) * 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 14:47 kharlan@deploy1003: kharlan: Continuing with deployment * 14:46 kharlan@deploy1003: kharlan: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.* * 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.* * 14:32 dkertesz: depooling cp7001{{!}}7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000) * 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof * 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:17 moritzm: installing Bird security updates * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie * 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.* * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s) * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie * 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] * 13:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s) * 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:35 moritzm: installing nghttp2 security updates * 13:35 awight@deploy1003: awight: Continuing with deployment * 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:33 awight@deploy1003: awight: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s) * 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment * 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki * 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.* * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:13 hnowlan: restarted restbase on restbase2029 * 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:50 hnowlan: deleting stuck mw-web pods in eqiad * 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) * 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision * 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] * 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) * 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision * 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] * 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) * 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw * 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:03 moritzm: installing apr-util security updates * 09:46 moritzm: installing bind9 security updates (client-side tools/libs only) * 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet * 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 * 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 * 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet * 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet * 09:01 XioNoX: asw1-b13-drmrs> request system reboot - [[phab:T437984|T437984]] * 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner * 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 * 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:58 moritzm: installing node-lodash security updates * 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 * 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision * 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 * 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 * 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade * 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade * 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet * 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:22 XioNoX: asw1-b12-drmrs> request system reboot - [[phab:T437984|T437984]] * 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 * 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 * 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner * 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade * 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade * 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production ([[phab:T437213|T437213]]) * 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply * 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production * 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply * 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw * 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw * 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green * 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here * 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected ([[phab:T439010|T439010]]) * 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet * 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected ([[phab:T439010|T439010]]) * 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected ([[phab:T439010|T439010]]) * 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts * 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though * 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) * 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state ([[phab:T439010|T439010]]) * 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* * 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-09-23 == * 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:56 brett: Switching acme-chief primary from codfw to eqiad - [[phab:T439010|T439010]] * 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones * 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia ([[phab:T438293|T438293]]) * 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. * 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 121m 40s) * 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next * 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next * 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected ([[phab:T439010|T439010]]) * 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see * 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected ([[phab:T439010|T439010]]) * 22:24 ryankemper: [Cirrus] s/expected/expect * 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel * 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) * 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status * 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 51m 29s) * 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 21:18 Emperor: ceph mgr fail on apus-be2005 * 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003 * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing * 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% * 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster * 20:49 swfrench@dns1004: END - running authdns-update * 20:46 swfrench@dns1004: START - running authdns-update * 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) * 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 20:06 dzahn@dns1004: END - running authdns-update * 20:03 dzahn@dns1004: START - running authdns-update * 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie * 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 19:33 volans: rebooting arclamp2001.codfw.wmnet * 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on * 19:17 taavi@dns1004: END - running authdns-update * 19:14 taavi@dns1004: START - running authdns-update * 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* * 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on * 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors * 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors * 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org * 18:45 sukhe@dns1004: END - running authdns-update * 18:43 sukhe@dns1004: START - running authdns-update * 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org * 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org * 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie * 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors * 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors * 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:01 cdanis@dns1005: END - running authdns-update * 17:58 cdanis@dns1005: START - running authdns-update * 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:54 taavi@dns1004: END - running authdns-update * 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:51 taavi@dns1004: START - running authdns-update * 17:46 taavi@dns1004: END - running authdns-update * 17:43 taavi@dns1004: START - running authdns-update * 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - [[phab:T439010|T439010]] * 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie * 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 * 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 * 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie * 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie * 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:54 moritzm: installing cjose security updates * 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* * 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) * 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:46 dancy@deploy1003: Started scap sync-world: testing * 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie * 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' * 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 [[phab:T438961|T438961]] * 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 * 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors * 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ [[phab:T438961|T438961]] * 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d [[phab:T438961|T438961]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before [[phab:T438786|T438786]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox * 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 [[phab:T438961|T438961]] * 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie * 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 * 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts * 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) * 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance * 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance * 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) * 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie * 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* * 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] * 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie * 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:56 awight: manually run puppet agent * 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:45 awight: add seanleong-wmde to deployment-prep * 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad * 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 12:22 kharlan@deploy1003: kharlan: Continuing with deployment * 12:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:01 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] * 11:51 Dreamy_Jazz: Deployed patch for [[phab:T438729|T438729]] * 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - [[phab:T438896|T438896]] * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web [[phab:T438896|T438896]] (duration: 04m 20s) * 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web [[phab:T438896|T438896]] * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:13 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] (duration: 12m 52s) * 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts * 08:04 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] * 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:59 moritzm: installing giflib security updates * 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:29 moritzm: installing python-idna security updates * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance * 00:46 ryankemper: [WDQS] [[phab:T435443|T435443]] Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture * 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance == 2026-09-22 == * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] (duration: 13m 43s) * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 21:53 rzl@deploy1003: rzl: Continuing with deployment * 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] * 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) * 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) * 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet * 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) * 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] * 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) * 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) * 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] * 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:40 sbisson@deploy1003: sbisson: Continuing with deployment * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 sbisson@deploy1003: sbisson: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:35 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] * 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] (duration: 13m 35s) * 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment * 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] * 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] (duration: 10m 29s) * 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.* * 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:11 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] * 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie * 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie * 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0 * 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts * 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s) * 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts * 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s) * 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0 * 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] (duration: 10m 09s) * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030 * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030 * 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030 * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] * 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030 * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029 * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029 * 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029 * 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet * 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:24 dzahn@dns1004: END - running authdns-update * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 17:22 dzahn@dns1004: START - running authdns-update * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed * 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] (duration: 13m 13s) * 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance * 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance * 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing * 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] * 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing * 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance * 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance * 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues * 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 15:50 moritzm: installing libhtml-parser-perl security updates * 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance * 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox * 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120 * 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120 * 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120 * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts * 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s) * 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s) * 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN [[phab:T436571|T436571]] * 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120 * 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002 * 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002 * 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 12:44 marostegui: Stop mariadb on db2250:s5 [[phab:T437411|T437411]] [[phab:T437279|T437279]] * 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] (duration: 33m 01s) * 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:08 kharlan@deploy1003: kharlan: Continuing with deployment * 12:06 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host (duration: 00m 54s) * 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host * 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host (duration: 01m 01s) * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host * 11:46 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] * 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] (duration: 13m 46s) * 11:34 kharlan@deploy1003: kharlan: Continuing with deployment * 11:33 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] * 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] (duration: 33m 09s) * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 11:13 kharlan@deploy1003: kharlan: Continuing with deployment * 11:12 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922 * 10:51 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] * 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie * 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 * 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 * 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 * 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 * 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 * 10:06 moritzm: installing libcap2 security updates * 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 * 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 * 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 * 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 * 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 * 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 * 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 * 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 * 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 * 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003 * 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 * 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues * 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues * 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie * 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - [[phab:T438828|T438828]] * 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues * 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues * 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations * 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet * 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:26 btullis@dns1004: END - running authdns-update * 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:23 btullis@dns1004: START - running authdns-update * 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet * 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 * 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 * 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3 * 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320 * 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320 * 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3 * 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1 * 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1 * 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:32 moritzm: installig zip security updates * 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:29 XioNoX: asw1-bw27-esams> request system reboot - [[phab:T437984|T437984]] * 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27 * 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27 * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:18 moritzm: installing gst-plugins-base1.0 security updates * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot * 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot * 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:23 awight: UTC morning deployment window complete * 07:22 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] (duration: 17m 46s) * 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment * 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] * 07:02 moritzm: installing pyasn1 security updates * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] (duration: 35m 52s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-21 == * 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:18 maryum: Deployed security fix for [[phab:T437708|T437708]] * 20:35 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] (duration: 15m 56s) * 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment * 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:19 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired * 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.* * 18:30 Emperor: repool eqiad sessionstore [[phab:T437915|T437915]] * 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired * 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie * 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm * 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie * 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.* * 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006 * 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie * 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie * 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] (duration: 12m 40s) * 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie * 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:36 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply * 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply * 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] * 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.* * 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie * 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.* * 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie * 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply * 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply * 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie * 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:13 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] (duration: 08m 09s) * 14:08 zabe@deploy1003: zabe: Continuing with deployment * 14:08 zabe@deploy1003: zabe: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie * 14:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 14:01 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] (duration: 12m 30s) * 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment * 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # [[phab:T438716|T438716]] * 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] (duration: 11m 40s) * 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment * 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] * 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] (duration: 14m 19s) * 13:15 mfossati@deploy1003: mfossati: Continuing with deployment * 13:14 mfossati@deploy1003: mfossati: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet * 13:07 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] * 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org * 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet * 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX) * 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet * 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet * 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet * 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # [[phab:T438499|T438499]] * 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet * 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie * 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:35 moritzm: installing chromium security updates * 09:22 tappof: bump space for prometheus k8s-dse in eqiad * 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s) * 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie * 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] * 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision * 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - [[phab:T438291|T438291]] * 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - [[phab:T438291|T438291]] * 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - [[phab:T438291|T438291]] * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw * 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet * 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003 * 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 [[phab:T435314|T435314]] * 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org == 2026-09-20 == * 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply == 2026-09-19 == * 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually ([[phab:T438591|T438591]], see [[phab:T438591|T438591]]#12341750 for what I did exactly) * 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> f7lyyypdwph7g4km2aii6h1cyo1giza 2466269 2466268 2026-10-10T12:00:13Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2466269 wikitext text/x-wiki == 2026-10-10 == * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-09 == * 18:54 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) * 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment * 18:48 cscott@deploy1003: cscott, ssastry: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:44 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] * 17:38 Daimona: Run database query from [[phab:T440557|T440557]]#12411029 in x1.wikishared * 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 * 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet * 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606{{!}}T440606]]' * 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet * 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet * 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet * 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet * 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location * 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet * 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet * 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet * 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet * 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test * 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test * 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test * 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm * 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A * 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch * 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump * 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm * 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet * 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet * 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet * 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) * 01:25 kemayo@deploy1003: kemayo: Continuing with deployment * 01:24 kemayo@deploy1003: kemayo: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 01:20 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] * 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr [[phab:T440068|T440068]] - downloads 120x times faster == 2026-10-08 == * 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning * 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) * 23:16 ejegg@deploy1003: ejegg: Continuing with deployment * 23:11 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:51 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] * 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) * 22:25 kemayo@deploy1003: kemayo: Continuing with deployment * 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 22:20 kemayo@deploy1003: kemayo: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 22:00 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] * 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) * 21:37 ejegg@deploy1003: ejegg: Rolling back deployment * 21:27 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad * 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:07 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] * 20:48 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) * 20:43 cjming@deploy1003: cjming: Continuing with deployment * 20:42 cjming@deploy1003: cjming: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:38 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] * 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] (duration: 09m 41s) * 20:27 mlitn@deploy1003: mlitn: Continuing with deployment * 20:27 mlitn@deploy1003: mlitn: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:23 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] * 20:21 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) * 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment * 20:10 cjming@deploy1003: milazg, cjming: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] * 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) * 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update * 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet * 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet * 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet * 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes * 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts * 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s) * 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad * 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change * 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie * 16:07 swfrench@dns1004: END - running authdns-update * 16:04 swfrench@dns1004: START - running authdns-update * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) * 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet * 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet * 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet * 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning * 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet * 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet * 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes * 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:54 cmooney@dns3003: END - running authdns-update * 14:51 cmooney@dns3003: START - running authdns-update * 14:50 cgoubert@dns2004: END - running authdns-update * 14:48 cgoubert@dns2004: START - running authdns-update * 14:48 moritzm: installing pyasn1 security updates * 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet * 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] (duration: 36m 09s) * 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet * 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 14:23 moritzm: installing perl security updates * 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet * 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 14:03 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:47 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) * 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet * 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment * 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet * 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet * 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet * 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet * 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet * 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # [[phab:T440023|T440023]] [[phab:T439902|T439902]] * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet * 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet * 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet * 12:30 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) * 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet * 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet * 12:25 zabe@deploy1003: zabe: Continuing with deployment * 12:24 zabe@deploy1003: zabe: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet * 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] * 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet * 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet * 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet * 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet * 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet * 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] (duration: 04m 15s) * 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet * 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet * 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet * 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] * 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:10 topranks: cr1-codfw switch back to master routing engine [[phab:T426197|T426197]] * 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet * 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:36 topranks: cr1-codfw switch to backup routing engine [[phab:T426197|T426197]] * 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host * 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet * 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet * 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet * 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover * 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet * 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet * 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host * 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade [[phab:T417871|T417871]] * 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet * 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet * 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet * 08:59 moritzm: installing Wireshark security updates * 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json * 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json * 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - [[phab:T436922|T436922]] * 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json * 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 [[phab:T436922|T436922]] * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet * 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet * 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:42 moritzm: installing python3.13 security updates * 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet * 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:01 moritzm: installing mbedtls security updates * 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet * 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet * 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) * 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet * 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment * 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet * 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet * 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet * 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet * 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups * 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet * 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet * 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet * 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet * 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old ([[phab:T440358|T440358]]) * 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet * 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet * 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet * 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet * 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet * 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet * 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet * 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet * 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet * 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet * 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet * 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet * 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet * 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet * 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet * 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet * 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet * 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet * 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet * 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet * 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet * 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet * 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet * 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet * 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet * 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet * 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet * 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet * 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet * 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet * 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet * 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet * 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font ([[phab:T393763|T393763]]) == 2026-10-07 == * 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm * 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet * 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) * 20:23 kemayo@deploy1003: kemayo: Continuing with deployment * 20:21 kemayo@deploy1003: kemayo: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] * 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply * 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply * 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts * 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s) * 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet * 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] (duration: 10m 40s) * 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] * 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet * 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet * 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for [[phab:T417800|T417800]] (duration: 05m 23s) * 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:17 blake@deploy1003: Started scap sync-world: add restart policies for [[phab:T417800|T417800]] * 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet * 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet * 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) * 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet * 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) * 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet * 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) * 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade * 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade * 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet * 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting * 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade * 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade * 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade * 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade * 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade * 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) * 14:48 jforrester@deploy1003: jforrester: Continuing with deployment * 14:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams * 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams * 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs * 14:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] * 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs * 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin * 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo * 14:34 jforrester@deploy1003: jforrester: Continuing with deployment * 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo * 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:27 moritzm: installing PHP 8.2 security updates * 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru * 14:26 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet * 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet * 14:22 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] * 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 14:19 moritzm: installing postgresql-15 security updates * 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - [[phab:T439840|T439840]] * 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet * 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet * 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment * 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:54 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet * 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 13:44 papaul: start maintenance on cr2-codfw * 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) * 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet * 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] * 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) * 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:13 moritzm: installing mbedtls security updates * 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) * 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet * 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply * 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet * 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet * 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet * 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet * 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet * 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet * 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet * 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet * 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 andre: Rolling back again wmf.23 from group1 to group0 - [[phab:T438219|T438219]] * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet * 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet * 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for [[phab:T417800|T417800]] (duration: 07m 32s) * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for [[phab:T417800|T417800]] * 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet * 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet * 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet * 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet * 09:54 duesen: just a test for SAL * 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:50 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) * 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet * 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:36 daniel@deploy1003: daniel: Continuing with deployment * 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet * 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet * 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:26 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet * 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet * 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet * 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet * 09:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:13 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] * 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) * 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] * 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) * 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] * 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet * 08:20 aklapper@deploy1003: Rolling back deployment * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet * 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet * 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet * 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet * 08:02 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) * 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment * 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet * 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia [[phab:T392465|T392465]] * 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:25 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] * 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet * 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet * 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet * 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 moritzm: installing bird security updates * 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet * 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet * 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet * 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet * 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet * 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet * 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet * 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:10 kart_: Updated Apertium to 2026-09-30-051826-production ([[phab:T437213|T437213]]) * 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply * 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply * 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply * 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply * 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host [[phab:T437279|T437279]] * 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning * 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host * 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply * 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-06 == * 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) * 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:00 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] * 20:15 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 * 20:10 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt * 20:03 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] * 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test * 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet * 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet * 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test * 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet * 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet * 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test * 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet * 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet * 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet * 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet * 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet * 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet * 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet * 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet * 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet * 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:39 bking@dns2004: END - running authdns-update * 16:37 bking@dns2004: START - running authdns-update * 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet * 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet * 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet * 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet * 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:26 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) * 15:21 daniel@deploy1003: daniel: Continuing with deployment * 15:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing python3.13 security updates * 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 15:11 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] * 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw * 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] (duration: 00m 42s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] * 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment * 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] (duration: 00m 51s) * 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] * 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment * 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:57 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet * 14:48 moritzm: restarting pybal on lvs1020 [[phab:T439730|T439730]] * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw * 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet * 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) * 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment * 14:33 moritzm: restarting pybal on lvs2012 [[phab:T439730|T439730]] * 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet * 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet * 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) * 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet * 14:23 moritzm: restarting pybal on lvs2014 [[phab:T439730|T439730]] * 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb * 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb * 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet * 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet * 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb * 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet * 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet * 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding chec}} * 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment * 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin * 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet * {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a s}} * 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet * 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding check}} * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment * 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] * 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 moritzm: installing php8.2 security updates * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 214m 10s) * 13:31 slyngshede@dns1004: END - running authdns-update * 13:29 slyngshede@dns1004: START - running authdns-update * 13:25 moritzm: installing libssh2 security updates * 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 13:14 moritzm: installing glibc security updates * 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 11:33 moritzm: installing chromium security updates * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply * 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply * 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 * 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back * 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor ([[phab:T406724|T406724]]) * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading * 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 [[phab:T436082|T436082]] * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 moritzm: installing curl security updates * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing * 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m {{!}} tee gc_run_06102026` - [[phab:T437297|T437297]] * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 * 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 00m 19s) * 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 * 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back * 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 * 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing * 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing * 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning * 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning * 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning * 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 * 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back * 08:00 matthiasmullie: UTC morning backport window done * 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) * 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:50 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] * 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) * 07:38 dcausse@deploy1003: dcausse: Continuing with deployment * 07:38 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 07:31 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] * 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 * 07:21 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) * 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back * 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment * 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] * 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 * 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 * 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning * 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 * 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) * 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] (duration: 27m 17s) * 03:28 mwpresync@deploy1003: Rolling back deployment * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-05 == * 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) * 23:18 derenrich@deploy1003: derenrich: Continuing with deployment * 23:16 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:12 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet * 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439719|T439719]] * 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439974|T439974]] * 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:51 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439932|T439932]] * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams * 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs * 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs * 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb) * 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # [[phab:T439577|T439577]] * 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) * 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes * 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply * 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply * 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru * 16:32 dcausse: closing the "Distributed search queries experiment" window * 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) * 16:25 dcausse@deploy1003: dcausse: Continuing with deployment * 16:23 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox * 16:19 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] * 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad * 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) * 15:59 dcausse@deploy1003: dcausse: Continuing with deployment * 15:58 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:54 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] * 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad * 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] (duration: 25m 12s) * 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm * 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet * 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox * 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:27 dcausse@deploy1003: dcausse: Continuing with deployment * 15:26 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet * 15:22 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm * 15:16 elukey@dns1004: END - running authdns-update * 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] (duration: 10m 11s) * 15:14 elukey@dns1004: START - running authdns-update * 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet * 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:10 dcausse@deploy1003: dcausse: Continuing with deployment * 15:09 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:05 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet * 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] (duration: 10m 12s) * 14:53 dcausse@deploy1003: dcausse: Continuing with deployment * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:52 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet * 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) * 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] (duration: 10m 04s) * 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet * 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet * 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 dcausse@deploy1003: dcausse: Continuing with deployment * 14:32 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:28 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] * 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) * 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:15 dcausse@deploy1003: dcausse: Continuing with deployment * 14:13 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:09 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439968|T439968]] * 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet * 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet * 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] (duration: 09m 59s) * 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service * 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:37 zabe@deploy1003: zabe: Continuing with deployment * 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:36 zabe@deploy1003: zabe: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] * 13:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) * 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet * 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet * 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet * 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:21 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] * 13:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet * 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet * 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment * 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet * 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet * 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet * 13:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet * 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet * 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet * 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie * 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) * 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:31 moritzm: installing sqlite3 security updates * 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment * 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:16 moritzm: installing mesa security updates * 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org * 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org * 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org * 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org * 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005 * 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm * 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet * 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] * 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet * 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet * 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet * 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet * 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet * 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:46 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] (duration: 04m 44s) * 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:42 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] * 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing * 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests * 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing * 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests * 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm * 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests * 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests * 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing * 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing * 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet * 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet * 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing * 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm * 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests * 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests * 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet * 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet * 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie * 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for [[phab:T417800|T417800]] (duration: 09m 40s) * 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for [[phab:T417800|T417800]] * 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet * 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet * 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:22 volans: upgrading cloudcumin2001 in-place to trixie [[phab:T436667|T436667]] * 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm * 07:49 dpogorzelski@dns1004: END - running authdns-update * 07:46 dpogorzelski@dns1004: START - running authdns-update * 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-04 == * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-03 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-02 == * 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` [[phab:T440033|T440033]] * 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - [[phab:T427353|T427353]] * 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie * 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet * 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet * 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet * 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet * 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet * 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet * 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet * 17:44 dzahn@dns1004: END - running authdns-update * 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet * 17:41 dzahn@dns1004: START - running authdns-update * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet * 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet * 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 * 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 * 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox * 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 * 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet * 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet * 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes * 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) * 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes * 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet * 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm * 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet * 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox * 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm * 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox * 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts * 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm * 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox * 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts * 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-10-01 == * 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.* * 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) * 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] * 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — [[phab:T432444|T432444]] * 21:24 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) * 21:19 egardner@deploy1003: egardner: Continuing with deployment * 21:17 egardner@deploy1003: egardner: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] * 21:06 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) * 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment * 20:59 brennen@deploy1003: brennen, srapoj: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] * 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - [[phab:T432444|T432444]] * 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) * 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:37 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] * 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) * 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment * 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:18 cjming@deploy1003: cjming, volker-e: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can * 20:14 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] * 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two * 19:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] (duration: 09m 40s) * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:46 zabe@deploy1003: zabe: Continuing with deployment * 19:46 zabe@deploy1003: zabe: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:42 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] * 19:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) * 19:36 zabe@deploy1003: zabe: Continuing with deployment * 19:36 zabe@deploy1003: zabe: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] * 19:19 jhathaway: puppet re-enabled * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 * 18:25 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): logs kind of a mess, but no current blockers, rolling to all wikis * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox * 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T439100|T439100]] (duration: 33m 25s) * 17:16 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T439100|T439100]] * 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:32 moritzm: installing socat security updates * 15:29 dpogorzelski@dns1004: END - running authdns-update * 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:25 dpogorzelski@dns1004: START - running authdns-update * 15:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] (duration: 10m 12s) * 15:14 zabe@deploy1003: zabe: Continuing with deployment * 15:14 zabe@deploy1003: zabe: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing postgresql security updates * 15:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] * 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.* * 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.* * 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release * 14:48 moritzm: installing busybox bugfix updates from trixie point release * 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) * 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment * 14:42 moritzm: installing zsh bugfix update from trixie point release * 14:41 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] * 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release * 14:31 dpogorzelski@dns1004: END - running authdns-update * 14:29 dpogorzelski@dns1004: START - running authdns-update * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 14:04 awight: EU afternoon backport complete * 14:03 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) * 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:58 awight@deploy1003: lilients, awight: Continuing with deployment * 13:57 awight@deploy1003: lilients, awight: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:53 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:41 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) * 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:36 awight@deploy1003: awight, migr: Continuing with deployment * 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 awight@deploy1003: awight, migr: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve * 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:18 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] * 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment * 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:34 moritzm: installing libtasn1-6 security updates * 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) * 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:22 moritzm: installing glibc security updates * 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] * 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) * 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] * 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad * 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) * 11:50 jforrester@deploy1003: jforrester: Continuing with deployment * 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad * 11:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] * 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw * 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw * 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) * 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment * 10:44 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # [[phab:T439805|T439805]] * 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) * 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 * 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates * 08:05 moritzm: installing pcre2/openssl security updates * 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o) * 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) * 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:16 kharlan@deploy1003: kharlan: Continuing with deployment * 07:15 kharlan@deploy1003: kharlan: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] * 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:54 kharlan@deploy1003: kharlan: Continuing with deployment * 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:53 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet * 06:33 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] * 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) * 06:23 kharlan@deploy1003: kharlan: Continuing with deployment * 06:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:18 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] == 2026-09-30 == * 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 21:22 sbassett: Deployed updated mitigation for [[phab:T435455|T435455]] * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:52 arlolra@deploy1003: arlolra: Continuing with deployment * 20:52 arlolra@deploy1003: arlolra: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:47 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] * 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment * 20:31 mstyles@deploy1003: mstyles, catrope: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] * 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage * 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod * 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:22 brennen@deploy1003: Rolling back deployment * 18:10 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): no current blockers, rolling to group1 * 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:15 bking@dns2004: END - running authdns-update * 17:13 bking@dns2004: START - running authdns-update * 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet * 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet * 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet * 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:54 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet * 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:49 zabe@deploy1003: zabe: Continuing with deployment * 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:48 zabe@deploy1003: zabe: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet * 14:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] * 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet * 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet * 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet * 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) * 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:30 jforrester@deploy1003: jforrester: Continuing with deployment * 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:29 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) * 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: jforrester: Continuing with deployment * 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:16 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] * 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] (duration: 11m 33s) * 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 Amir1: mwscript-k8s --comment="[[phab:T358266|T358266]]" --local_dblist=<(expanddblist all {{!}} grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing ([[phab:T358266|T358266]]) * 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary [[phab:T437937|T437937]] * 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment * 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] * 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) * 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment * 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet * 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet * 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet * 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment * 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:25 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet * 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] * 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet * 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet * 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment * 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:10 samtar@deploy1003: migr, samtar: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] * 13:05 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] (duration: 10m 41s) * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet * 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet * 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) * 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - [[phab:T439667|T439667]] * 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet * 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:26 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] * 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet * 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet * 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet * 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 11:07 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] (duration: 11m 12s) * 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment * 10:59 cgoubert@deploy1003: cgoubert: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:56 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet * 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - [[phab:T438454|T438454]] * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:08 btullis@dns1004: END - running authdns-update * 09:06 btullis@dns1004: START - running authdns-update * 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' * 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' * 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad * 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad * 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca * 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw * 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw * 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:46 elukey@dns1004: END - running authdns-update * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:44 elukey@dns1004: START - running authdns-update * 07:35 moritzm: installing pcre2/openssl security updates * 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] (duration: 11m 27s) * 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:09 dcausse@deploy1003: dcausse: Continuing with deployment * 07:07 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] * 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet * 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet * 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-29 == * 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 * 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 * 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 * 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox * 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 * 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 * 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - [[phab:T427353|T427353]] [[phab:T395938|T395938]] * 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) * 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment * 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] * 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) * 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 * 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 * 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:43 cklimas@deploy1003: cklimas: Continuing with deployment * 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:38 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:33 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] * 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox * 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 * 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) * 20:19 sbassett@deploy1003: sbassett: Rolling back deployment * 20:12 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm * 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:08 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet * 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 18:06 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]) - no current blockers, rolling to group0 * 18:05 sandraebele: Deployed Refinery as part of weekly deployment train * 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) * 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] * 17:53 sandraebele: Deploying Refinery as part of weekly deployment train * 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) * 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] * 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie * 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train * 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 * 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` [[phab:T436186|T436186]] * 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # [[phab:T418194|T418194]] * 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply * 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply * 15:59 jhathaway: updating puppet stdlib, should be a noop * 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 14:21 moritzm: installing rsyslog security updates * 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet * 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet * 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw * 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:05 krinkle@deploy1003: krinkle: Continuing with deployment * 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox * 14:00 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 13:56 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] * 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] (duration: 00m 09s) * 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) * 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] * 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:25 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) * 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:20 samtar@deploy1003: samtar: Continuing with deployment * 13:19 samtar@deploy1003: samtar: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:14 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] * 13:10 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) * 13:05 samtar@deploy1003: samtar: Continuing with deployment * 13:04 samtar@deploy1003: samtar: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:00 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] * 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging * 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) * 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment * 12:44 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] * 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — [[phab:T432444|T432444]] * 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 * 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for [[phab:T417800|T417800]] (duration: 04m 14s) * 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for [[phab:T417800|T417800]] * 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox * 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet * 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # [[phab:T418194|T418194]] * 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) * 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment * 08:52 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] * 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad * 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad * 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad * 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:10 moritzm: installing PHP 8.4 security updates * 07:49 matthiasmullie: UTC morning backport window done * 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) * 07:44 mlitn@deploy1003: mlitn: Continuing with deployment * 07:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:39 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] * 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) * 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 07:30 kharlan@deploy1003: kharlan: Continuing with deployment * 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 07:27 kharlan@deploy1003: kharlan: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:26 kart_: Updated cxserver to 2026-09-25-114231-production * 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 07:23 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] * 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org * 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) * 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org * 07:11 mlitn@deploy1003: mlitn: Continuing with deployment * 07:09 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now * 07:04 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] * 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply * 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply * 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm * 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm * 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm * 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm * 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm * 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure * 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 * 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033 * 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] (duration: 36m 03s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie == 2026-09-28 == * 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:35 zabe: run script to fix fr_metadata drifts # [[phab:T420341|T420341]] * 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 21:47 sbassett: Deployed security mitigation for [[phab:T435455|T435455]] * 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync * 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync * 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - [[phab:T439201|T439201]] * 21:13 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) * 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment * 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 cjming@deploy1003: volker-e, cjming: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:00 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] * 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) * 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia [[phab:T438375|T438375]] * 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 20:46 kemayo@deploy1003: kemayo: Continuing with deployment * 20:45 kemayo@deploy1003: kemayo: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] * 20:32 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) * 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie * 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment * 20:26 cjming@deploy1003: ssastry, cjming: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:22 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] * 20:16 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) * 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment * 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.* * 20:07 aude@deploy1003: aude, volker-e: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:03 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie * 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts * 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s) * 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... * 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts * 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s) * 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie * 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie * 16:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) * 16:15 zabe@deploy1003: zabe: Continuing with deployment * 16:10 zabe@deploy1003: zabe: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:06 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] * 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] (duration: 19m 54s) * 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # [[phab:T439319|T439319]] * 15:45 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw * 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw * 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 14:55 Lucas_WMDE: UTC afternoon backport+config window done * 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) * 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment * 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) * 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] * 13:58 vgutierrez: repool cp7001 - [[phab:T439361|T439361]] * 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) * 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment * 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - [[phab:T432444|T432444]] * 13:47 bking@dns2004: END - running authdns-update * 13:44 bking@dns2004: START - running authdns-update * 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] * 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw * 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) * 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 vgutierrez: depool cp7001 for some purged tests - [[phab:T439361|T439361]] * 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw * 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw * 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] * 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) * 13:11 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] * 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet * 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:28 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) * 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:22 zabe@deploy1003: zabe: Continuing with deployment * 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet * 11:21 zabe@deploy1003: zabe: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] * 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 10:25 moritzm: installing chromium security updates * 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet * 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet * 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor * 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:16 moritzm: installing Exim security updates * 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:59 matthiasmullie: UTC morning backport window complete * 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) * 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment * 07:49 mlitn@deploy1003: migr, mlitn: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:44 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] * 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] (duration: 32m 06s) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:27 mlitn@deploy1003: mlitn: Continuing with deployment * 07:26 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi * 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] * 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-27 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-26 == * 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) * 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 * 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] (duration: 10m 53s) * 08:02 oblivian@deploy1003: oblivian: Continuing with deployment * 08:00 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:56 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] * 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] (duration: 11m 01s) * 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 03:30 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-25 == * 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values (duration: 00m 11s) * 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values * 17:59 cdanis@dns1004: END - running authdns-update * 17:57 cdanis@dns1004: START - running authdns-update * 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie * 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet * 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - [[phab:T343000|T343000]] * 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* * 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # [[phab:T438750|T438750]] * 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet * 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active * 13:24 Emperor: repool sessionstore in codfw * 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet * 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox * 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet * 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet * 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # [[phab:T438366|T438366]] * 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet * 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - [[phab:T439105|T439105]] * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet * 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet * 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox * 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet * 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.* * 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.* * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet == 2026-09-24 == * 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate * {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on t}} * 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment * 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment}} * {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on te}} * 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.* * 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet * 21:45 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s) * 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:40 catrope@deploy1003: catrope: Continuing with deployment * 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet * 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet * 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:33 catrope@deploy1003: catrope: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:28 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] * 21:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] (duration: 19m 27s) * 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment * 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi * 21:08 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] * 21:04 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s) * 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.* * 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.* * 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment * 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:53 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] * 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:49 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s) * 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:44 catrope@deploy1003: catrope: Continuing with deployment * 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:43 catrope@deploy1003: catrope: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet * 20:39 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s) * 20:14 kemayo@deploy1003: kemayo: Continuing with deployment * 20:12 kemayo@deploy1003: kemayo: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 20:08 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] * 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance * 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro * 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet` * 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet * 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness * 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance * 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly * 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance * 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw * 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance * 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters * 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian{{!}}tegola-vector-tiles) * 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s) * 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment * 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] * 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance * 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance * 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance * 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance * 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw * 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance * 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance * 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance * 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance * 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability * 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s) * 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.* * 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw * 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.* * 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift * 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master{{!}}device-analytics{{!}}echostore{{!}}helm-charts{{!}}k8s-ingress-wikikube-ro{{!}}linkrecommendation{{!}}mathoid{{!}}restbase{{!}}restbase-async{{!}}rest-gateway-ro{{!}}mobileapps{{!}}mwdebug.*{{!}}push-notifications{{!}}recommendation-api{{!}}releases{{!}}wikifeeds * 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid{{!}}zotero * 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium{{!}}schema{{!}}termbox{{!}}proton{{!}}cxserver * 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor * 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.* * 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.* * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro * 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions * 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s) * 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo * 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet * 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie * 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.* * 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes * 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet * 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] * 15:35 vgutierrez@dns1004: END - running authdns-update * 15:33 vgutierrez@dns1004: START - running authdns-update * 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s) * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet * 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie * 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox * 15:06 vgutierrez@dns1004: END - running authdns-update * 15:04 vgutierrez@dns1004: START - running authdns-update * 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update * 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s) * 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.* * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.* * 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:51 dkertesz: repooling cp7001{{!}}7011 after successful testing ([[phab:T343000|T343000]]) * 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 14:47 kharlan@deploy1003: kharlan: Continuing with deployment * 14:46 kharlan@deploy1003: kharlan: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.* * 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.* * 14:32 dkertesz: depooling cp7001{{!}}7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000) * 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof * 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:17 moritzm: installing Bird security updates * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie * 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.* * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s) * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie * 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] * 13:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s) * 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:35 moritzm: installing nghttp2 security updates * 13:35 awight@deploy1003: awight: Continuing with deployment * 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:33 awight@deploy1003: awight: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s) * 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment * 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki * 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.* * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:13 hnowlan: restarted restbase on restbase2029 * 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:50 hnowlan: deleting stuck mw-web pods in eqiad * 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) * 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision * 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] * 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) * 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision * 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] * 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) * 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw * 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:03 moritzm: installing apr-util security updates * 09:46 moritzm: installing bind9 security updates (client-side tools/libs only) * 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet * 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 * 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 * 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet * 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet * 09:01 XioNoX: asw1-b13-drmrs> request system reboot - [[phab:T437984|T437984]] * 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner * 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 * 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:58 moritzm: installing node-lodash security updates * 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 * 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision * 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 * 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 * 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade * 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade * 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet * 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:22 XioNoX: asw1-b12-drmrs> request system reboot - [[phab:T437984|T437984]] * 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 * 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 * 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner * 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade * 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade * 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production ([[phab:T437213|T437213]]) * 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply * 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production * 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply * 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw * 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw * 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green * 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here * 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected ([[phab:T439010|T439010]]) * 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet * 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected ([[phab:T439010|T439010]]) * 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected ([[phab:T439010|T439010]]) * 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts * 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though * 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) * 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state ([[phab:T439010|T439010]]) * 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* * 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-09-23 == * 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:56 brett: Switching acme-chief primary from codfw to eqiad - [[phab:T439010|T439010]] * 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones * 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia ([[phab:T438293|T438293]]) * 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. * 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 121m 40s) * 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next * 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next * 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected ([[phab:T439010|T439010]]) * 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see * 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected ([[phab:T439010|T439010]]) * 22:24 ryankemper: [Cirrus] s/expected/expect * 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel * 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) * 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status * 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 51m 29s) * 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 21:18 Emperor: ceph mgr fail on apus-be2005 * 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003 * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing * 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% * 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster * 20:49 swfrench@dns1004: END - running authdns-update * 20:46 swfrench@dns1004: START - running authdns-update * 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) * 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 20:06 dzahn@dns1004: END - running authdns-update * 20:03 dzahn@dns1004: START - running authdns-update * 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie * 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 19:33 volans: rebooting arclamp2001.codfw.wmnet * 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on * 19:17 taavi@dns1004: END - running authdns-update * 19:14 taavi@dns1004: START - running authdns-update * 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* * 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on * 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors * 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors * 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org * 18:45 sukhe@dns1004: END - running authdns-update * 18:43 sukhe@dns1004: START - running authdns-update * 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org * 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org * 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie * 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors * 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors * 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:01 cdanis@dns1005: END - running authdns-update * 17:58 cdanis@dns1005: START - running authdns-update * 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:54 taavi@dns1004: END - running authdns-update * 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:51 taavi@dns1004: START - running authdns-update * 17:46 taavi@dns1004: END - running authdns-update * 17:43 taavi@dns1004: START - running authdns-update * 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - [[phab:T439010|T439010]] * 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie * 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 * 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 * 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie * 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie * 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:54 moritzm: installing cjose security updates * 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* * 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) * 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:46 dancy@deploy1003: Started scap sync-world: testing * 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie * 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' * 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 [[phab:T438961|T438961]] * 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 * 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors * 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ [[phab:T438961|T438961]] * 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d [[phab:T438961|T438961]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before [[phab:T438786|T438786]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox * 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 [[phab:T438961|T438961]] * 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie * 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 * 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts * 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) * 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance * 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance * 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) * 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie * 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* * 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] * 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie * 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:56 awight: manually run puppet agent * 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:45 awight: add seanleong-wmde to deployment-prep * 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad * 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 12:22 kharlan@deploy1003: kharlan: Continuing with deployment * 12:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:01 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] * 11:51 Dreamy_Jazz: Deployed patch for [[phab:T438729|T438729]] * 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - [[phab:T438896|T438896]] * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web [[phab:T438896|T438896]] (duration: 04m 20s) * 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web [[phab:T438896|T438896]] * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:13 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] (duration: 12m 52s) * 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts * 08:04 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] * 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:59 moritzm: installing giflib security updates * 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:29 moritzm: installing python-idna security updates * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance * 00:46 ryankemper: [WDQS] [[phab:T435443|T435443]] Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture * 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance == 2026-09-22 == * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] (duration: 13m 43s) * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 21:53 rzl@deploy1003: rzl: Continuing with deployment * 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] * 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) * 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) * 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet * 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) * 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] * 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) * 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) * 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] * 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:40 sbisson@deploy1003: sbisson: Continuing with deployment * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 sbisson@deploy1003: sbisson: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:35 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] * 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] (duration: 13m 35s) * 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment * 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] * 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] (duration: 10m 29s) * 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.* * 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:11 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] * 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie * 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie * 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0 * 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts * 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s) * 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts * 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s) * 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0 * 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] (duration: 10m 09s) * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030 * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030 * 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030 * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] * 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030 * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029 * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029 * 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029 * 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet * 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:24 dzahn@dns1004: END - running authdns-update * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 17:22 dzahn@dns1004: START - running authdns-update * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed * 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] (duration: 13m 13s) * 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance * 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance * 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing * 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] * 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing * 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance * 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance * 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues * 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 15:50 moritzm: installing libhtml-parser-perl security updates * 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance * 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox * 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120 * 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120 * 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120 * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts * 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s) * 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s) * 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN [[phab:T436571|T436571]] * 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120 * 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002 * 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002 * 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 12:44 marostegui: Stop mariadb on db2250:s5 [[phab:T437411|T437411]] [[phab:T437279|T437279]] * 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] (duration: 33m 01s) * 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:08 kharlan@deploy1003: kharlan: Continuing with deployment * 12:06 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host (duration: 00m 54s) * 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host * 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host (duration: 01m 01s) * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host * 11:46 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] * 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] (duration: 13m 46s) * 11:34 kharlan@deploy1003: kharlan: Continuing with deployment * 11:33 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] * 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] (duration: 33m 09s) * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 11:13 kharlan@deploy1003: kharlan: Continuing with deployment * 11:12 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922 * 10:51 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] * 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie * 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 * 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 * 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 * 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 * 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 * 10:06 moritzm: installing libcap2 security updates * 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 * 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 * 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 * 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 * 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 * 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 * 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 * 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 * 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 * 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003 * 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 * 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues * 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues * 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie * 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - [[phab:T438828|T438828]] * 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues * 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues * 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations * 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet * 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:26 btullis@dns1004: END - running authdns-update * 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:23 btullis@dns1004: START - running authdns-update * 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet * 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 * 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 * 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3 * 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320 * 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320 * 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3 * 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1 * 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1 * 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:32 moritzm: installig zip security updates * 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:29 XioNoX: asw1-bw27-esams> request system reboot - [[phab:T437984|T437984]] * 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27 * 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27 * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:18 moritzm: installing gst-plugins-base1.0 security updates * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot * 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot * 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:23 awight: UTC morning deployment window complete * 07:22 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] (duration: 17m 46s) * 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment * 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] * 07:02 moritzm: installing pyasn1 security updates * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] (duration: 35m 52s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-21 == * 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:18 maryum: Deployed security fix for [[phab:T437708|T437708]] * 20:35 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] (duration: 15m 56s) * 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment * 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:19 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired * 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.* * 18:30 Emperor: repool eqiad sessionstore [[phab:T437915|T437915]] * 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired * 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie * 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm * 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie * 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.* * 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006 * 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie * 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie * 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] (duration: 12m 40s) * 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie * 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:36 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply * 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply * 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] * 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.* * 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie * 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.* * 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie * 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply * 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply * 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie * 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:13 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] (duration: 08m 09s) * 14:08 zabe@deploy1003: zabe: Continuing with deployment * 14:08 zabe@deploy1003: zabe: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie * 14:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 14:01 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] (duration: 12m 30s) * 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment * 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # [[phab:T438716|T438716]] * 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] (duration: 11m 40s) * 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment * 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] * 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] (duration: 14m 19s) * 13:15 mfossati@deploy1003: mfossati: Continuing with deployment * 13:14 mfossati@deploy1003: mfossati: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet * 13:07 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] * 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org * 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet * 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX) * 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet * 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet * 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet * 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # [[phab:T438499|T438499]] * 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet * 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie * 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:35 moritzm: installing chromium security updates * 09:22 tappof: bump space for prometheus k8s-dse in eqiad * 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s) * 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie * 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] * 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision * 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - [[phab:T438291|T438291]] * 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - [[phab:T438291|T438291]] * 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - [[phab:T438291|T438291]] * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw * 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet * 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003 * 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 [[phab:T435314|T435314]] * 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org == 2026-09-20 == * 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply == 2026-09-19 == * 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually ([[phab:T438591|T438591]], see [[phab:T438591|T438591]]#12341750 for what I did exactly) * 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> k90aac9bqz5gkx0h42kbawabf5ld1ce 2466279 2466269 2026-10-11T00:00:07Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync 2466279 wikitext text/x-wiki == 2026-10-11 == * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-10 == * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-09 == * 18:54 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) * 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment * 18:48 cscott@deploy1003: cscott, ssastry: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:44 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] * 17:38 Daimona: Run database query from [[phab:T440557|T440557]]#12411029 in x1.wikishared * 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 * 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet * 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606{{!}}T440606]]' * 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet * 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet * 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet * 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet * 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location * 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet * 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet * 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet * 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet * 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test * 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test * 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test * 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm * 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A * 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch * 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump * 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm * 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet * 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet * 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet * 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) * 01:25 kemayo@deploy1003: kemayo: Continuing with deployment * 01:24 kemayo@deploy1003: kemayo: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 01:20 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] * 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr [[phab:T440068|T440068]] - downloads 120x times faster == 2026-10-08 == * 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning * 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) * 23:16 ejegg@deploy1003: ejegg: Continuing with deployment * 23:11 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:51 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] * 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) * 22:25 kemayo@deploy1003: kemayo: Continuing with deployment * 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 22:20 kemayo@deploy1003: kemayo: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 22:00 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] * 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) * 21:37 ejegg@deploy1003: ejegg: Rolling back deployment * 21:27 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad * 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:07 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] * 20:48 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) * 20:43 cjming@deploy1003: cjming: Continuing with deployment * 20:42 cjming@deploy1003: cjming: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:38 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] * 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] (duration: 09m 41s) * 20:27 mlitn@deploy1003: mlitn: Continuing with deployment * 20:27 mlitn@deploy1003: mlitn: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:23 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] * 20:21 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) * 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment * 20:10 cjming@deploy1003: milazg, cjming: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] * 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) * 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update * 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet * 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet * 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet * 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes * 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts * 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s) * 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad * 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change * 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie * 16:07 swfrench@dns1004: END - running authdns-update * 16:04 swfrench@dns1004: START - running authdns-update * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) * 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet * 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet * 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet * 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning * 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet * 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet * 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes * 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:54 cmooney@dns3003: END - running authdns-update * 14:51 cmooney@dns3003: START - running authdns-update * 14:50 cgoubert@dns2004: END - running authdns-update * 14:48 cgoubert@dns2004: START - running authdns-update * 14:48 moritzm: installing pyasn1 security updates * 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet * 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] (duration: 36m 09s) * 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet * 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 14:23 moritzm: installing perl security updates * 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet * 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 14:03 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:47 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) * 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet * 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment * 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet * 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet * 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet * 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet * 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet * 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # [[phab:T440023|T440023]] [[phab:T439902|T439902]] * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet * 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet * 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet * 12:30 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) * 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet * 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet * 12:25 zabe@deploy1003: zabe: Continuing with deployment * 12:24 zabe@deploy1003: zabe: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet * 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] * 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet * 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet * 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet * 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet * 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet * 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] (duration: 04m 15s) * 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet * 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet * 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet * 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] * 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:10 topranks: cr1-codfw switch back to master routing engine [[phab:T426197|T426197]] * 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet * 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:36 topranks: cr1-codfw switch to backup routing engine [[phab:T426197|T426197]] * 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host * 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet * 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet * 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet * 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover * 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet * 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet * 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host * 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade [[phab:T417871|T417871]] * 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet * 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet * 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet * 08:59 moritzm: installing Wireshark security updates * 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json * 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json * 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - [[phab:T436922|T436922]] * 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json * 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 [[phab:T436922|T436922]] * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet * 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet * 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:42 moritzm: installing python3.13 security updates * 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet * 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:01 moritzm: installing mbedtls security updates * 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet * 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet * 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) * 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet * 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment * 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet * 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet * 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet * 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet * 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups * 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet * 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet * 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet * 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet * 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old ([[phab:T440358|T440358]]) * 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet * 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet * 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet * 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet * 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet * 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet * 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet * 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet * 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet * 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet * 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet * 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet * 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet * 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet * 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet * 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet * 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet * 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet * 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet * 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet * 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet * 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet * 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet * 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet * 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet * 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet * 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet * 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet * 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet * 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet * 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet * 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet * 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font ([[phab:T393763|T393763]]) == 2026-10-07 == * 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm * 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet * 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) * 20:23 kemayo@deploy1003: kemayo: Continuing with deployment * 20:21 kemayo@deploy1003: kemayo: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] * 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply * 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply * 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts * 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s) * 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet * 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] (duration: 10m 40s) * 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] * 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet * 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet * 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for [[phab:T417800|T417800]] (duration: 05m 23s) * 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:17 blake@deploy1003: Started scap sync-world: add restart policies for [[phab:T417800|T417800]] * 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet * 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet * 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) * 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet * 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) * 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet * 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) * 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade * 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade * 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet * 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting * 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade * 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade * 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade * 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade * 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade * 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) * 14:48 jforrester@deploy1003: jforrester: Continuing with deployment * 14:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams * 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams * 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs * 14:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] * 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs * 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin * 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo * 14:34 jforrester@deploy1003: jforrester: Continuing with deployment * 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo * 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:27 moritzm: installing PHP 8.2 security updates * 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru * 14:26 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet * 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet * 14:22 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] * 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 14:19 moritzm: installing postgresql-15 security updates * 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - [[phab:T439840|T439840]] * 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet * 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet * 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment * 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:54 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet * 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 13:44 papaul: start maintenance on cr2-codfw * 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) * 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet * 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] * 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) * 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:13 moritzm: installing mbedtls security updates * 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) * 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet * 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply * 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet * 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet * 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet * 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet * 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet * 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet * 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet * 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet * 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 andre: Rolling back again wmf.23 from group1 to group0 - [[phab:T438219|T438219]] * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet * 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet * 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for [[phab:T417800|T417800]] (duration: 07m 32s) * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for [[phab:T417800|T417800]] * 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet * 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet * 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet * 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet * 09:54 duesen: just a test for SAL * 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:50 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) * 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet * 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:36 daniel@deploy1003: daniel: Continuing with deployment * 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet * 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet * 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:26 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet * 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet * 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet * 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet * 09:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:13 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] * 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) * 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] * 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) * 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] * 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet * 08:20 aklapper@deploy1003: Rolling back deployment * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet * 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet * 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet * 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet * 08:02 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) * 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment * 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet * 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia [[phab:T392465|T392465]] * 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:25 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] * 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet * 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet * 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet * 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 moritzm: installing bird security updates * 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet * 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet * 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet * 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet * 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet * 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet * 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet * 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:10 kart_: Updated Apertium to 2026-09-30-051826-production ([[phab:T437213|T437213]]) * 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply * 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply * 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply * 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply * 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host [[phab:T437279|T437279]] * 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning * 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host * 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply * 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-06 == * 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) * 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:00 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] * 20:15 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 * 20:10 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt * 20:03 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] * 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test * 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet * 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet * 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test * 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet * 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet * 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test * 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet * 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet * 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet * 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet * 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet * 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet * 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet * 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet * 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet * 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:39 bking@dns2004: END - running authdns-update * 16:37 bking@dns2004: START - running authdns-update * 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet * 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet * 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet * 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet * 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:26 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) * 15:21 daniel@deploy1003: daniel: Continuing with deployment * 15:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing python3.13 security updates * 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 15:11 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] * 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw * 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] (duration: 00m 42s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] * 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment * 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] (duration: 00m 51s) * 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] * 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment * 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:57 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet * 14:48 moritzm: restarting pybal on lvs1020 [[phab:T439730|T439730]] * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw * 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet * 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) * 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment * 14:33 moritzm: restarting pybal on lvs2012 [[phab:T439730|T439730]] * 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet * 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet * 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) * 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet * 14:23 moritzm: restarting pybal on lvs2014 [[phab:T439730|T439730]] * 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb * 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb * 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet * 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet * 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb * 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet * 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet * 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding chec}} * 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment * 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin * 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet * {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a s}} * 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet * 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding check}} * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment * 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] * 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 moritzm: installing php8.2 security updates * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 214m 10s) * 13:31 slyngshede@dns1004: END - running authdns-update * 13:29 slyngshede@dns1004: START - running authdns-update * 13:25 moritzm: installing libssh2 security updates * 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 13:14 moritzm: installing glibc security updates * 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 11:33 moritzm: installing chromium security updates * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply * 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply * 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 * 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back * 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor ([[phab:T406724|T406724]]) * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading * 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 [[phab:T436082|T436082]] * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 moritzm: installing curl security updates * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing * 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m {{!}} tee gc_run_06102026` - [[phab:T437297|T437297]] * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 * 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 00m 19s) * 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 * 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back * 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 * 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing * 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing * 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning * 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning * 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning * 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 * 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back * 08:00 matthiasmullie: UTC morning backport window done * 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) * 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:50 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] * 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) * 07:38 dcausse@deploy1003: dcausse: Continuing with deployment * 07:38 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 07:31 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] * 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 * 07:21 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) * 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back * 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment * 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] * 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 * 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 * 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning * 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 * 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) * 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] (duration: 27m 17s) * 03:28 mwpresync@deploy1003: Rolling back deployment * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-05 == * 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) * 23:18 derenrich@deploy1003: derenrich: Continuing with deployment * 23:16 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:12 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet * 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439719|T439719]] * 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439974|T439974]] * 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:51 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439932|T439932]] * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams * 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs * 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs * 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb) * 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # [[phab:T439577|T439577]] * 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) * 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes * 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply * 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply * 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru * 16:32 dcausse: closing the "Distributed search queries experiment" window * 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) * 16:25 dcausse@deploy1003: dcausse: Continuing with deployment * 16:23 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox * 16:19 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] * 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad * 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) * 15:59 dcausse@deploy1003: dcausse: Continuing with deployment * 15:58 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:54 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] * 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad * 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] (duration: 25m 12s) * 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm * 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet * 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox * 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:27 dcausse@deploy1003: dcausse: Continuing with deployment * 15:26 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet * 15:22 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm * 15:16 elukey@dns1004: END - running authdns-update * 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] (duration: 10m 11s) * 15:14 elukey@dns1004: START - running authdns-update * 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet * 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:10 dcausse@deploy1003: dcausse: Continuing with deployment * 15:09 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:05 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet * 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] (duration: 10m 12s) * 14:53 dcausse@deploy1003: dcausse: Continuing with deployment * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:52 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet * 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) * 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] (duration: 10m 04s) * 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet * 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet * 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 dcausse@deploy1003: dcausse: Continuing with deployment * 14:32 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:28 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] * 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) * 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:15 dcausse@deploy1003: dcausse: Continuing with deployment * 14:13 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:09 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439968|T439968]] * 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet * 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet * 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] (duration: 09m 59s) * 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service * 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:37 zabe@deploy1003: zabe: Continuing with deployment * 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:36 zabe@deploy1003: zabe: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] * 13:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) * 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet * 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet * 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet * 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:21 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] * 13:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet * 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet * 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment * 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet * 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet * 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet * 13:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet * 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet * 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet * 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie * 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) * 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:31 moritzm: installing sqlite3 security updates * 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment * 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:16 moritzm: installing mesa security updates * 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org * 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org * 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org * 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org * 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005 * 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm * 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet * 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] * 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet * 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet * 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet * 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet * 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet * 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:46 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] (duration: 04m 44s) * 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:42 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] * 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing * 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests * 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing * 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests * 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm * 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests * 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests * 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing * 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing * 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet * 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet * 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing * 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm * 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests * 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests * 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet * 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet * 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie * 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for [[phab:T417800|T417800]] (duration: 09m 40s) * 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for [[phab:T417800|T417800]] * 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet * 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet * 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:22 volans: upgrading cloudcumin2001 in-place to trixie [[phab:T436667|T436667]] * 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm * 07:49 dpogorzelski@dns1004: END - running authdns-update * 07:46 dpogorzelski@dns1004: START - running authdns-update * 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-04 == * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-03 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-02 == * 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` [[phab:T440033|T440033]] * 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - [[phab:T427353|T427353]] * 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie * 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet * 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet * 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet * 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet * 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet * 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet * 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet * 17:44 dzahn@dns1004: END - running authdns-update * 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet * 17:41 dzahn@dns1004: START - running authdns-update * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet * 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet * 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 * 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 * 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox * 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 * 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet * 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet * 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes * 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) * 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes * 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet * 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm * 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet * 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox * 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm * 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox * 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts * 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm * 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox * 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts * 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-10-01 == * 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.* * 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) * 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] * 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — [[phab:T432444|T432444]] * 21:24 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) * 21:19 egardner@deploy1003: egardner: Continuing with deployment * 21:17 egardner@deploy1003: egardner: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] * 21:06 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) * 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment * 20:59 brennen@deploy1003: brennen, srapoj: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] * 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - [[phab:T432444|T432444]] * 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) * 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:37 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] * 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) * 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment * 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:18 cjming@deploy1003: cjming, volker-e: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can * 20:14 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] * 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two * 19:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] (duration: 09m 40s) * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:46 zabe@deploy1003: zabe: Continuing with deployment * 19:46 zabe@deploy1003: zabe: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:42 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] * 19:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) * 19:36 zabe@deploy1003: zabe: Continuing with deployment * 19:36 zabe@deploy1003: zabe: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] * 19:19 jhathaway: puppet re-enabled * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 * 18:25 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): logs kind of a mess, but no current blockers, rolling to all wikis * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox * 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T439100|T439100]] (duration: 33m 25s) * 17:16 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T439100|T439100]] * 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:32 moritzm: installing socat security updates * 15:29 dpogorzelski@dns1004: END - running authdns-update * 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:25 dpogorzelski@dns1004: START - running authdns-update * 15:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] (duration: 10m 12s) * 15:14 zabe@deploy1003: zabe: Continuing with deployment * 15:14 zabe@deploy1003: zabe: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing postgresql security updates * 15:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] * 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.* * 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.* * 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release * 14:48 moritzm: installing busybox bugfix updates from trixie point release * 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) * 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment * 14:42 moritzm: installing zsh bugfix update from trixie point release * 14:41 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] * 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release * 14:31 dpogorzelski@dns1004: END - running authdns-update * 14:29 dpogorzelski@dns1004: START - running authdns-update * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 14:04 awight: EU afternoon backport complete * 14:03 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) * 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:58 awight@deploy1003: lilients, awight: Continuing with deployment * 13:57 awight@deploy1003: lilients, awight: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:53 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:41 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) * 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:36 awight@deploy1003: awight, migr: Continuing with deployment * 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 awight@deploy1003: awight, migr: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve * 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:18 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] * 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment * 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:34 moritzm: installing libtasn1-6 security updates * 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) * 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:22 moritzm: installing glibc security updates * 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] * 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) * 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] * 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad * 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) * 11:50 jforrester@deploy1003: jforrester: Continuing with deployment * 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad * 11:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] * 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw * 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw * 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) * 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment * 10:44 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # [[phab:T439805|T439805]] * 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) * 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 * 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates * 08:05 moritzm: installing pcre2/openssl security updates * 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o) * 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) * 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:16 kharlan@deploy1003: kharlan: Continuing with deployment * 07:15 kharlan@deploy1003: kharlan: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] * 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:54 kharlan@deploy1003: kharlan: Continuing with deployment * 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:53 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet * 06:33 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] * 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) * 06:23 kharlan@deploy1003: kharlan: Continuing with deployment * 06:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:18 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] == 2026-09-30 == * 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 21:22 sbassett: Deployed updated mitigation for [[phab:T435455|T435455]] * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:52 arlolra@deploy1003: arlolra: Continuing with deployment * 20:52 arlolra@deploy1003: arlolra: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:47 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] * 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment * 20:31 mstyles@deploy1003: mstyles, catrope: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] * 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage * 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod * 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:22 brennen@deploy1003: Rolling back deployment * 18:10 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): no current blockers, rolling to group1 * 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:15 bking@dns2004: END - running authdns-update * 17:13 bking@dns2004: START - running authdns-update * 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet * 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet * 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet * 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:54 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet * 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:49 zabe@deploy1003: zabe: Continuing with deployment * 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:48 zabe@deploy1003: zabe: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet * 14:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] * 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet * 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet * 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet * 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) * 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:30 jforrester@deploy1003: jforrester: Continuing with deployment * 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:29 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) * 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: jforrester: Continuing with deployment * 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:16 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] * 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] (duration: 11m 33s) * 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 Amir1: mwscript-k8s --comment="[[phab:T358266|T358266]]" --local_dblist=<(expanddblist all {{!}} grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing ([[phab:T358266|T358266]]) * 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary [[phab:T437937|T437937]] * 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment * 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] * 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) * 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment * 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet * 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet * 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet * 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment * 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:25 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet * 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] * 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet * 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet * 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment * 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:10 samtar@deploy1003: migr, samtar: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] * 13:05 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] (duration: 10m 41s) * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet * 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet * 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) * 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - [[phab:T439667|T439667]] * 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet * 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:26 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] * 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet * 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet * 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet * 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 11:07 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] (duration: 11m 12s) * 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment * 10:59 cgoubert@deploy1003: cgoubert: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:56 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet * 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - [[phab:T438454|T438454]] * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:08 btullis@dns1004: END - running authdns-update * 09:06 btullis@dns1004: START - running authdns-update * 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' * 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' * 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad * 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad * 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca * 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw * 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw * 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:46 elukey@dns1004: END - running authdns-update * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:44 elukey@dns1004: START - running authdns-update * 07:35 moritzm: installing pcre2/openssl security updates * 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] (duration: 11m 27s) * 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:09 dcausse@deploy1003: dcausse: Continuing with deployment * 07:07 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] * 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet * 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet * 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-29 == * 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 * 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 * 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 * 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox * 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 * 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 * 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - [[phab:T427353|T427353]] [[phab:T395938|T395938]] * 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) * 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment * 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] * 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) * 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 * 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 * 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:43 cklimas@deploy1003: cklimas: Continuing with deployment * 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:38 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:33 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] * 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox * 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 * 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) * 20:19 sbassett@deploy1003: sbassett: Rolling back deployment * 20:12 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm * 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:08 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet * 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 18:06 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]) - no current blockers, rolling to group0 * 18:05 sandraebele: Deployed Refinery as part of weekly deployment train * 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) * 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] * 17:53 sandraebele: Deploying Refinery as part of weekly deployment train * 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) * 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] * 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie * 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train * 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 * 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` [[phab:T436186|T436186]] * 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # [[phab:T418194|T418194]] * 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply * 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply * 15:59 jhathaway: updating puppet stdlib, should be a noop * 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 14:21 moritzm: installing rsyslog security updates * 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet * 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet * 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw * 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:05 krinkle@deploy1003: krinkle: Continuing with deployment * 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox * 14:00 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 13:56 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] * 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] (duration: 00m 09s) * 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) * 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] * 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:25 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) * 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:20 samtar@deploy1003: samtar: Continuing with deployment * 13:19 samtar@deploy1003: samtar: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:14 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] * 13:10 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) * 13:05 samtar@deploy1003: samtar: Continuing with deployment * 13:04 samtar@deploy1003: samtar: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:00 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] * 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging * 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) * 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment * 12:44 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] * 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — [[phab:T432444|T432444]] * 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 * 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for [[phab:T417800|T417800]] (duration: 04m 14s) * 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for [[phab:T417800|T417800]] * 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox * 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet * 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # [[phab:T418194|T418194]] * 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) * 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment * 08:52 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] * 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad * 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad * 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad * 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:10 moritzm: installing PHP 8.4 security updates * 07:49 matthiasmullie: UTC morning backport window done * 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) * 07:44 mlitn@deploy1003: mlitn: Continuing with deployment * 07:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:39 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] * 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) * 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 07:30 kharlan@deploy1003: kharlan: Continuing with deployment * 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 07:27 kharlan@deploy1003: kharlan: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:26 kart_: Updated cxserver to 2026-09-25-114231-production * 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 07:23 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] * 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org * 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) * 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org * 07:11 mlitn@deploy1003: mlitn: Continuing with deployment * 07:09 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now * 07:04 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] * 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply * 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply * 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm * 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm * 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm * 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm * 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm * 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure * 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 * 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033 * 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] (duration: 36m 03s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie == 2026-09-28 == * 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:35 zabe: run script to fix fr_metadata drifts # [[phab:T420341|T420341]] * 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 21:47 sbassett: Deployed security mitigation for [[phab:T435455|T435455]] * 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync * 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync * 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - [[phab:T439201|T439201]] * 21:13 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) * 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment * 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 cjming@deploy1003: volker-e, cjming: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:00 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] * 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) * 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia [[phab:T438375|T438375]] * 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 20:46 kemayo@deploy1003: kemayo: Continuing with deployment * 20:45 kemayo@deploy1003: kemayo: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] * 20:32 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) * 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie * 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment * 20:26 cjming@deploy1003: ssastry, cjming: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:22 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] * 20:16 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) * 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment * 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.* * 20:07 aude@deploy1003: aude, volker-e: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:03 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie * 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts * 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s) * 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... * 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts * 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s) * 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie * 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie * 16:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) * 16:15 zabe@deploy1003: zabe: Continuing with deployment * 16:10 zabe@deploy1003: zabe: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:06 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] * 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] (duration: 19m 54s) * 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # [[phab:T439319|T439319]] * 15:45 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw * 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw * 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 14:55 Lucas_WMDE: UTC afternoon backport+config window done * 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) * 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment * 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) * 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] * 13:58 vgutierrez: repool cp7001 - [[phab:T439361|T439361]] * 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) * 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment * 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - [[phab:T432444|T432444]] * 13:47 bking@dns2004: END - running authdns-update * 13:44 bking@dns2004: START - running authdns-update * 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] * 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw * 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) * 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 vgutierrez: depool cp7001 for some purged tests - [[phab:T439361|T439361]] * 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw * 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw * 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] * 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) * 13:11 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] * 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet * 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:28 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) * 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:22 zabe@deploy1003: zabe: Continuing with deployment * 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet * 11:21 zabe@deploy1003: zabe: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] * 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 10:25 moritzm: installing chromium security updates * 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet * 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet * 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor * 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:16 moritzm: installing Exim security updates * 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:59 matthiasmullie: UTC morning backport window complete * 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) * 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment * 07:49 mlitn@deploy1003: migr, mlitn: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:44 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] * 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] (duration: 32m 06s) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:27 mlitn@deploy1003: mlitn: Continuing with deployment * 07:26 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi * 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] * 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-27 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-26 == * 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) * 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 * 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] (duration: 10m 53s) * 08:02 oblivian@deploy1003: oblivian: Continuing with deployment * 08:00 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:56 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] * 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] (duration: 11m 01s) * 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 03:30 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-25 == * 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values (duration: 00m 11s) * 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values * 17:59 cdanis@dns1004: END - running authdns-update * 17:57 cdanis@dns1004: START - running authdns-update * 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie * 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet * 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - [[phab:T343000|T343000]] * 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* * 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # [[phab:T438750|T438750]] * 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet * 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active * 13:24 Emperor: repool sessionstore in codfw * 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet * 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox * 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet * 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet * 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # [[phab:T438366|T438366]] * 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet * 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - [[phab:T439105|T439105]] * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet * 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet * 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox * 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet * 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.* * 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.* * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet == 2026-09-24 == * 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate * {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on t}} * 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment * 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment}} * {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on te}} * 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.* * 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet * 21:45 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s) * 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:40 catrope@deploy1003: catrope: Continuing with deployment * 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet * 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet * 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:33 catrope@deploy1003: catrope: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:28 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] * 21:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] (duration: 19m 27s) * 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment * 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi * 21:08 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] * 21:04 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s) * 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.* * 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.* * 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment * 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:53 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] * 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:49 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s) * 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:44 catrope@deploy1003: catrope: Continuing with deployment * 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:43 catrope@deploy1003: catrope: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet * 20:39 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s) * 20:14 kemayo@deploy1003: kemayo: Continuing with deployment * 20:12 kemayo@deploy1003: kemayo: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 20:08 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] * 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance * 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro * 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet` * 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet * 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness * 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance * 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly * 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance * 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw * 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance * 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters * 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian{{!}}tegola-vector-tiles) * 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s) * 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment * 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] * 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance * 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance * 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance * 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance * 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw * 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance * 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance * 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance * 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance * 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability * 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s) * 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.* * 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw * 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.* * 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift * 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master{{!}}device-analytics{{!}}echostore{{!}}helm-charts{{!}}k8s-ingress-wikikube-ro{{!}}linkrecommendation{{!}}mathoid{{!}}restbase{{!}}restbase-async{{!}}rest-gateway-ro{{!}}mobileapps{{!}}mwdebug.*{{!}}push-notifications{{!}}recommendation-api{{!}}releases{{!}}wikifeeds * 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid{{!}}zotero * 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium{{!}}schema{{!}}termbox{{!}}proton{{!}}cxserver * 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor * 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.* * 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.* * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro * 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions * 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s) * 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo * 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet * 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie * 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.* * 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes * 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet * 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] * 15:35 vgutierrez@dns1004: END - running authdns-update * 15:33 vgutierrez@dns1004: START - running authdns-update * 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s) * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet * 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie * 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox * 15:06 vgutierrez@dns1004: END - running authdns-update * 15:04 vgutierrez@dns1004: START - running authdns-update * 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update * 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s) * 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.* * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.* * 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:51 dkertesz: repooling cp7001{{!}}7011 after successful testing ([[phab:T343000|T343000]]) * 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 14:47 kharlan@deploy1003: kharlan: Continuing with deployment * 14:46 kharlan@deploy1003: kharlan: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.* * 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.* * 14:32 dkertesz: depooling cp7001{{!}}7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000) * 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof * 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:17 moritzm: installing Bird security updates * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie * 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.* * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s) * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie * 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] * 13:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s) * 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:35 moritzm: installing nghttp2 security updates * 13:35 awight@deploy1003: awight: Continuing with deployment * 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:33 awight@deploy1003: awight: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s) * 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment * 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki * 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.* * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:13 hnowlan: restarted restbase on restbase2029 * 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:50 hnowlan: deleting stuck mw-web pods in eqiad * 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) * 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision * 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] * 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) * 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision * 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] * 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) * 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw * 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:03 moritzm: installing apr-util security updates * 09:46 moritzm: installing bind9 security updates (client-side tools/libs only) * 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet * 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 * 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 * 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet * 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet * 09:01 XioNoX: asw1-b13-drmrs> request system reboot - [[phab:T437984|T437984]] * 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner * 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 * 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:58 moritzm: installing node-lodash security updates * 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 * 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision * 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 * 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 * 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade * 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade * 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet * 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:22 XioNoX: asw1-b12-drmrs> request system reboot - [[phab:T437984|T437984]] * 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 * 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 * 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner * 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade * 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade * 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production ([[phab:T437213|T437213]]) * 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply * 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production * 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply * 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw * 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw * 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green * 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here * 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected ([[phab:T439010|T439010]]) * 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet * 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected ([[phab:T439010|T439010]]) * 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected ([[phab:T439010|T439010]]) * 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts * 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though * 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) * 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state ([[phab:T439010|T439010]]) * 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* * 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-09-23 == * 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:56 brett: Switching acme-chief primary from codfw to eqiad - [[phab:T439010|T439010]] * 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones * 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia ([[phab:T438293|T438293]]) * 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. * 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 121m 40s) * 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next * 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next * 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected ([[phab:T439010|T439010]]) * 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see * 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected ([[phab:T439010|T439010]]) * 22:24 ryankemper: [Cirrus] s/expected/expect * 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel * 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) * 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status * 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 51m 29s) * 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 21:18 Emperor: ceph mgr fail on apus-be2005 * 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003 * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing * 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% * 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster * 20:49 swfrench@dns1004: END - running authdns-update * 20:46 swfrench@dns1004: START - running authdns-update * 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) * 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 20:06 dzahn@dns1004: END - running authdns-update * 20:03 dzahn@dns1004: START - running authdns-update * 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie * 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 19:33 volans: rebooting arclamp2001.codfw.wmnet * 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on * 19:17 taavi@dns1004: END - running authdns-update * 19:14 taavi@dns1004: START - running authdns-update * 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* * 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on * 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors * 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors * 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org * 18:45 sukhe@dns1004: END - running authdns-update * 18:43 sukhe@dns1004: START - running authdns-update * 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org * 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org * 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie * 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors * 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors * 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:01 cdanis@dns1005: END - running authdns-update * 17:58 cdanis@dns1005: START - running authdns-update * 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:54 taavi@dns1004: END - running authdns-update * 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:51 taavi@dns1004: START - running authdns-update * 17:46 taavi@dns1004: END - running authdns-update * 17:43 taavi@dns1004: START - running authdns-update * 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - [[phab:T439010|T439010]] * 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie * 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 * 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 * 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie * 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie * 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:54 moritzm: installing cjose security updates * 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* * 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) * 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:46 dancy@deploy1003: Started scap sync-world: testing * 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie * 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' * 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 [[phab:T438961|T438961]] * 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 * 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors * 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ [[phab:T438961|T438961]] * 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d [[phab:T438961|T438961]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before [[phab:T438786|T438786]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox * 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 [[phab:T438961|T438961]] * 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie * 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 * 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts * 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) * 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance * 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance * 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) * 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie * 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* * 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] * 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie * 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:56 awight: manually run puppet agent * 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:45 awight: add seanleong-wmde to deployment-prep * 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad * 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 12:22 kharlan@deploy1003: kharlan: Continuing with deployment * 12:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:01 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] * 11:51 Dreamy_Jazz: Deployed patch for [[phab:T438729|T438729]] * 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - [[phab:T438896|T438896]] * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web [[phab:T438896|T438896]] (duration: 04m 20s) * 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web [[phab:T438896|T438896]] * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:13 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] (duration: 12m 52s) * 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts * 08:04 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] * 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:59 moritzm: installing giflib security updates * 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:29 moritzm: installing python-idna security updates * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance * 00:46 ryankemper: [WDQS] [[phab:T435443|T435443]] Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture * 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance == 2026-09-22 == * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] (duration: 13m 43s) * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 21:53 rzl@deploy1003: rzl: Continuing with deployment * 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] * 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) * 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) * 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet * 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) * 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] * 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) * 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) * 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] * 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:40 sbisson@deploy1003: sbisson: Continuing with deployment * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 sbisson@deploy1003: sbisson: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:35 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] * 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] (duration: 13m 35s) * 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment * 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] * 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] (duration: 10m 29s) * 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.* * 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:11 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] * 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie * 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie * 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0 * 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts * 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s) * 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts * 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s) * 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0 * 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] (duration: 10m 09s) * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030 * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030 * 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030 * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] * 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030 * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029 * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029 * 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029 * 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet * 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:24 dzahn@dns1004: END - running authdns-update * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 17:22 dzahn@dns1004: START - running authdns-update * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed * 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] (duration: 13m 13s) * 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance * 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance * 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing * 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] * 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing * 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance * 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance * 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues * 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 15:50 moritzm: installing libhtml-parser-perl security updates * 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance * 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox * 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120 * 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120 * 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120 * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts * 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s) * 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s) * 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN [[phab:T436571|T436571]] * 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120 * 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002 * 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002 * 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 12:44 marostegui: Stop mariadb on db2250:s5 [[phab:T437411|T437411]] [[phab:T437279|T437279]] * 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] (duration: 33m 01s) * 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:08 kharlan@deploy1003: kharlan: Continuing with deployment * 12:06 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host (duration: 00m 54s) * 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host * 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host (duration: 01m 01s) * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host * 11:46 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] * 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] (duration: 13m 46s) * 11:34 kharlan@deploy1003: kharlan: Continuing with deployment * 11:33 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] * 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] (duration: 33m 09s) * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 11:13 kharlan@deploy1003: kharlan: Continuing with deployment * 11:12 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922 * 10:51 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] * 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie * 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 * 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 * 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 * 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 * 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 * 10:06 moritzm: installing libcap2 security updates * 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 * 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 * 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 * 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 * 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 * 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 * 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 * 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 * 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 * 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003 * 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 * 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues * 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues * 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie * 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - [[phab:T438828|T438828]] * 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues * 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues * 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations * 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet * 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:26 btullis@dns1004: END - running authdns-update * 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:23 btullis@dns1004: START - running authdns-update * 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet * 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 * 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 * 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3 * 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320 * 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320 * 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3 * 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1 * 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1 * 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:32 moritzm: installig zip security updates * 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:29 XioNoX: asw1-bw27-esams> request system reboot - [[phab:T437984|T437984]] * 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27 * 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27 * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:18 moritzm: installing gst-plugins-base1.0 security updates * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot * 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot * 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:23 awight: UTC morning deployment window complete * 07:22 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] (duration: 17m 46s) * 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment * 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] * 07:02 moritzm: installing pyasn1 security updates * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] (duration: 35m 52s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-21 == * 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:18 maryum: Deployed security fix for [[phab:T437708|T437708]] * 20:35 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] (duration: 15m 56s) * 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment * 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:19 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired * 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.* * 18:30 Emperor: repool eqiad sessionstore [[phab:T437915|T437915]] * 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired * 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie * 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm * 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie * 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.* * 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006 * 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie * 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie * 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] (duration: 12m 40s) * 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie * 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:36 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply * 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply * 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] * 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.* * 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie * 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.* * 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie * 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply * 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply * 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie * 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:13 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] (duration: 08m 09s) * 14:08 zabe@deploy1003: zabe: Continuing with deployment * 14:08 zabe@deploy1003: zabe: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie * 14:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 14:01 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] (duration: 12m 30s) * 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment * 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # [[phab:T438716|T438716]] * 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] (duration: 11m 40s) * 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment * 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] * 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] (duration: 14m 19s) * 13:15 mfossati@deploy1003: mfossati: Continuing with deployment * 13:14 mfossati@deploy1003: mfossati: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet * 13:07 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] * 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org * 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet * 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX) * 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet * 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet * 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet * 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # [[phab:T438499|T438499]] * 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet * 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie * 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:35 moritzm: installing chromium security updates * 09:22 tappof: bump space for prometheus k8s-dse in eqiad * 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s) * 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie * 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] * 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision * 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - [[phab:T438291|T438291]] * 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - [[phab:T438291|T438291]] * 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - [[phab:T438291|T438291]] * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw * 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet * 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003 * 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 [[phab:T435314|T435314]] * 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org == 2026-09-20 == * 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply == 2026-09-19 == * 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually ([[phab:T438591|T438591]], see [[phab:T438591|T438591]]#12341750 for what I did exactly) * 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> rfa6s17e8y69sq42569bw30zkd7a4oa 2466280 2466279 2026-10-11T00:00:15Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync 2466280 wikitext text/x-wiki == 2026-10-11 == * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-10 == * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-09 == * 18:54 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) * 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment * 18:48 cscott@deploy1003: cscott, ssastry: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:44 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] * 17:38 Daimona: Run database query from [[phab:T440557|T440557]]#12411029 in x1.wikishared * 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 * 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet * 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606{{!}}T440606]]' * 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet * 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet * 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet * 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet * 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location * 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet * 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet * 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet * 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet * 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test * 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test * 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test * 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm * 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A * 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch * 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump * 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm * 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet * 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet * 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet * 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) * 01:25 kemayo@deploy1003: kemayo: Continuing with deployment * 01:24 kemayo@deploy1003: kemayo: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 01:20 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] * 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr [[phab:T440068|T440068]] - downloads 120x times faster == 2026-10-08 == * 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning * 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) * 23:16 ejegg@deploy1003: ejegg: Continuing with deployment * 23:11 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:51 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] * 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) * 22:25 kemayo@deploy1003: kemayo: Continuing with deployment * 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 22:20 kemayo@deploy1003: kemayo: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 22:00 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] * 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) * 21:37 ejegg@deploy1003: ejegg: Rolling back deployment * 21:27 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad * 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:07 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] * 20:48 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) * 20:43 cjming@deploy1003: cjming: Continuing with deployment * 20:42 cjming@deploy1003: cjming: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:38 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] * 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] (duration: 09m 41s) * 20:27 mlitn@deploy1003: mlitn: Continuing with deployment * 20:27 mlitn@deploy1003: mlitn: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:23 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] * 20:21 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) * 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment * 20:10 cjming@deploy1003: milazg, cjming: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] * 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) * 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update * 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet * 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet * 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet * 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes * 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts * 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s) * 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad * 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change * 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie * 16:07 swfrench@dns1004: END - running authdns-update * 16:04 swfrench@dns1004: START - running authdns-update * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) * 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet * 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet * 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet * 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning * 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet * 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet * 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes * 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:54 cmooney@dns3003: END - running authdns-update * 14:51 cmooney@dns3003: START - running authdns-update * 14:50 cgoubert@dns2004: END - running authdns-update * 14:48 cgoubert@dns2004: START - running authdns-update * 14:48 moritzm: installing pyasn1 security updates * 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet * 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] (duration: 36m 09s) * 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet * 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 14:23 moritzm: installing perl security updates * 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet * 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 14:03 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:47 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) * 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet * 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment * 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet * 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet * 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet * 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet * 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet * 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # [[phab:T440023|T440023]] [[phab:T439902|T439902]] * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet * 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet * 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet * 12:30 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) * 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet * 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet * 12:25 zabe@deploy1003: zabe: Continuing with deployment * 12:24 zabe@deploy1003: zabe: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet * 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] * 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet * 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet * 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet * 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet * 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet * 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] (duration: 04m 15s) * 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet * 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet * 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet * 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] * 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:10 topranks: cr1-codfw switch back to master routing engine [[phab:T426197|T426197]] * 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet * 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:36 topranks: cr1-codfw switch to backup routing engine [[phab:T426197|T426197]] * 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host * 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet * 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet * 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet * 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover * 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet * 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet * 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host * 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade [[phab:T417871|T417871]] * 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet * 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet * 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet * 08:59 moritzm: installing Wireshark security updates * 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json * 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json * 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - [[phab:T436922|T436922]] * 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json * 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 [[phab:T436922|T436922]] * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet * 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet * 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:42 moritzm: installing python3.13 security updates * 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet * 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:01 moritzm: installing mbedtls security updates * 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet * 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet * 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) * 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet * 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment * 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet * 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet * 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet * 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet * 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups * 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet * 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet * 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet * 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet * 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old ([[phab:T440358|T440358]]) * 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet * 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet * 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet * 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet * 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet * 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet * 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet * 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet * 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet * 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet * 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet * 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet * 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet * 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet * 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet * 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet * 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet * 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet * 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet * 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet * 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet * 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet * 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet * 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet * 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet * 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet * 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet * 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet * 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet * 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet * 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet * 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet * 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font ([[phab:T393763|T393763]]) == 2026-10-07 == * 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm * 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet * 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) * 20:23 kemayo@deploy1003: kemayo: Continuing with deployment * 20:21 kemayo@deploy1003: kemayo: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] * 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply * 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply * 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts * 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s) * 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet * 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] (duration: 10m 40s) * 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] * 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet * 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet * 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for [[phab:T417800|T417800]] (duration: 05m 23s) * 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:17 blake@deploy1003: Started scap sync-world: add restart policies for [[phab:T417800|T417800]] * 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet * 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet * 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) * 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet * 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) * 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet * 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) * 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade * 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade * 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet * 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting * 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade * 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade * 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade * 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade * 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade * 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) * 14:48 jforrester@deploy1003: jforrester: Continuing with deployment * 14:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams * 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams * 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs * 14:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] * 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs * 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin * 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo * 14:34 jforrester@deploy1003: jforrester: Continuing with deployment * 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo * 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:27 moritzm: installing PHP 8.2 security updates * 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru * 14:26 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet * 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet * 14:22 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] * 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 14:19 moritzm: installing postgresql-15 security updates * 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - [[phab:T439840|T439840]] * 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet * 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet * 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment * 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:54 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet * 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 13:44 papaul: start maintenance on cr2-codfw * 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) * 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet * 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] * 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) * 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:13 moritzm: installing mbedtls security updates * 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) * 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet * 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply * 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet * 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet * 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet * 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet * 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet * 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet * 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet * 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet * 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 andre: Rolling back again wmf.23 from group1 to group0 - [[phab:T438219|T438219]] * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet * 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet * 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for [[phab:T417800|T417800]] (duration: 07m 32s) * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for [[phab:T417800|T417800]] * 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet * 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet * 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet * 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet * 09:54 duesen: just a test for SAL * 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:50 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) * 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet * 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:36 daniel@deploy1003: daniel: Continuing with deployment * 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet * 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet * 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:26 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet * 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet * 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet * 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet * 09:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:13 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] * 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) * 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] * 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) * 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] * 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet * 08:20 aklapper@deploy1003: Rolling back deployment * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet * 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet * 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet * 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet * 08:02 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) * 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment * 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet * 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia [[phab:T392465|T392465]] * 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:25 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] * 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet * 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet * 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet * 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 moritzm: installing bird security updates * 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet * 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet * 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet * 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet * 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet * 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet * 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet * 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:10 kart_: Updated Apertium to 2026-09-30-051826-production ([[phab:T437213|T437213]]) * 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply * 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply * 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply * 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply * 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host [[phab:T437279|T437279]] * 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning * 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host * 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply * 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-06 == * 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) * 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:00 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] * 20:15 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 * 20:10 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt * 20:03 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] * 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test * 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet * 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet * 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test * 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet * 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet * 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test * 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet * 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet * 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet * 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet * 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet * 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet * 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet * 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet * 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet * 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:39 bking@dns2004: END - running authdns-update * 16:37 bking@dns2004: START - running authdns-update * 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet * 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet * 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet * 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet * 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:26 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) * 15:21 daniel@deploy1003: daniel: Continuing with deployment * 15:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing python3.13 security updates * 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 15:11 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] * 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw * 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] (duration: 00m 42s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] * 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment * 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] (duration: 00m 51s) * 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] * 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment * 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:57 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet * 14:48 moritzm: restarting pybal on lvs1020 [[phab:T439730|T439730]] * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw * 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet * 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) * 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment * 14:33 moritzm: restarting pybal on lvs2012 [[phab:T439730|T439730]] * 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet * 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet * 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) * 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet * 14:23 moritzm: restarting pybal on lvs2014 [[phab:T439730|T439730]] * 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb * 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb * 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet * 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet * 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb * 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet * 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet * 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding chec}} * 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment * 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin * 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet * {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a s}} * 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet * 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding check}} * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment * 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] * 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 moritzm: installing php8.2 security updates * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 214m 10s) * 13:31 slyngshede@dns1004: END - running authdns-update * 13:29 slyngshede@dns1004: START - running authdns-update * 13:25 moritzm: installing libssh2 security updates * 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 13:14 moritzm: installing glibc security updates * 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 11:33 moritzm: installing chromium security updates * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply * 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply * 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 * 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back * 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor ([[phab:T406724|T406724]]) * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading * 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 [[phab:T436082|T436082]] * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 moritzm: installing curl security updates * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing * 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m {{!}} tee gc_run_06102026` - [[phab:T437297|T437297]] * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 * 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 00m 19s) * 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 * 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back * 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 * 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing * 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing * 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning * 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning * 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning * 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 * 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back * 08:00 matthiasmullie: UTC morning backport window done * 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) * 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:50 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] * 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) * 07:38 dcausse@deploy1003: dcausse: Continuing with deployment * 07:38 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 07:31 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] * 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 * 07:21 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) * 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back * 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment * 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] * 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 * 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 * 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning * 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 * 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) * 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] (duration: 27m 17s) * 03:28 mwpresync@deploy1003: Rolling back deployment * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-05 == * 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) * 23:18 derenrich@deploy1003: derenrich: Continuing with deployment * 23:16 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:12 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet * 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439719|T439719]] * 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439974|T439974]] * 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:51 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439932|T439932]] * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams * 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs * 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs * 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb) * 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # [[phab:T439577|T439577]] * 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) * 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes * 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply * 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply * 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru * 16:32 dcausse: closing the "Distributed search queries experiment" window * 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) * 16:25 dcausse@deploy1003: dcausse: Continuing with deployment * 16:23 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox * 16:19 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] * 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad * 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) * 15:59 dcausse@deploy1003: dcausse: Continuing with deployment * 15:58 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:54 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] * 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad * 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] (duration: 25m 12s) * 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm * 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet * 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox * 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:27 dcausse@deploy1003: dcausse: Continuing with deployment * 15:26 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet * 15:22 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm * 15:16 elukey@dns1004: END - running authdns-update * 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] (duration: 10m 11s) * 15:14 elukey@dns1004: START - running authdns-update * 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet * 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:10 dcausse@deploy1003: dcausse: Continuing with deployment * 15:09 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:05 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet * 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] (duration: 10m 12s) * 14:53 dcausse@deploy1003: dcausse: Continuing with deployment * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:52 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet * 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) * 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] (duration: 10m 04s) * 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet * 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet * 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 dcausse@deploy1003: dcausse: Continuing with deployment * 14:32 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:28 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] * 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) * 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:15 dcausse@deploy1003: dcausse: Continuing with deployment * 14:13 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:09 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439968|T439968]] * 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet * 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet * 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] (duration: 09m 59s) * 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service * 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:37 zabe@deploy1003: zabe: Continuing with deployment * 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:36 zabe@deploy1003: zabe: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] * 13:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) * 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet * 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet * 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet * 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:21 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] * 13:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet * 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet * 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment * 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet * 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet * 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet * 13:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet * 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet * 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet * 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie * 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) * 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:31 moritzm: installing sqlite3 security updates * 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment * 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:16 moritzm: installing mesa security updates * 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org * 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org * 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org * 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org * 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005 * 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm * 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet * 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] * 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet * 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet * 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet * 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet * 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet * 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:46 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] (duration: 04m 44s) * 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:42 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] * 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing * 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests * 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing * 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests * 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm * 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests * 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests * 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing * 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing * 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet * 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet * 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing * 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm * 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests * 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests * 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet * 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet * 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie * 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for [[phab:T417800|T417800]] (duration: 09m 40s) * 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for [[phab:T417800|T417800]] * 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet * 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet * 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:22 volans: upgrading cloudcumin2001 in-place to trixie [[phab:T436667|T436667]] * 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm * 07:49 dpogorzelski@dns1004: END - running authdns-update * 07:46 dpogorzelski@dns1004: START - running authdns-update * 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-04 == * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-03 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-02 == * 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` [[phab:T440033|T440033]] * 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - [[phab:T427353|T427353]] * 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie * 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet * 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet * 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet * 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet * 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet * 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet * 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet * 17:44 dzahn@dns1004: END - running authdns-update * 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet * 17:41 dzahn@dns1004: START - running authdns-update * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet * 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet * 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 * 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 * 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox * 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 * 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet * 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet * 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes * 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) * 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes * 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet * 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm * 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet * 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox * 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm * 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox * 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts * 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm * 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox * 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts * 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-10-01 == * 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.* * 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) * 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] * 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — [[phab:T432444|T432444]] * 21:24 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) * 21:19 egardner@deploy1003: egardner: Continuing with deployment * 21:17 egardner@deploy1003: egardner: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] * 21:06 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) * 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment * 20:59 brennen@deploy1003: brennen, srapoj: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] * 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - [[phab:T432444|T432444]] * 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) * 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:37 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] * 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) * 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment * 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:18 cjming@deploy1003: cjming, volker-e: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can * 20:14 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] * 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two * 19:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] (duration: 09m 40s) * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:46 zabe@deploy1003: zabe: Continuing with deployment * 19:46 zabe@deploy1003: zabe: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:42 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] * 19:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) * 19:36 zabe@deploy1003: zabe: Continuing with deployment * 19:36 zabe@deploy1003: zabe: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] * 19:19 jhathaway: puppet re-enabled * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 * 18:25 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): logs kind of a mess, but no current blockers, rolling to all wikis * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox * 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T439100|T439100]] (duration: 33m 25s) * 17:16 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T439100|T439100]] * 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:32 moritzm: installing socat security updates * 15:29 dpogorzelski@dns1004: END - running authdns-update * 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:25 dpogorzelski@dns1004: START - running authdns-update * 15:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] (duration: 10m 12s) * 15:14 zabe@deploy1003: zabe: Continuing with deployment * 15:14 zabe@deploy1003: zabe: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing postgresql security updates * 15:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] * 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.* * 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.* * 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release * 14:48 moritzm: installing busybox bugfix updates from trixie point release * 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) * 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment * 14:42 moritzm: installing zsh bugfix update from trixie point release * 14:41 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] * 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release * 14:31 dpogorzelski@dns1004: END - running authdns-update * 14:29 dpogorzelski@dns1004: START - running authdns-update * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 14:04 awight: EU afternoon backport complete * 14:03 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) * 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:58 awight@deploy1003: lilients, awight: Continuing with deployment * 13:57 awight@deploy1003: lilients, awight: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:53 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:41 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) * 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:36 awight@deploy1003: awight, migr: Continuing with deployment * 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 awight@deploy1003: awight, migr: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve * 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:18 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] * 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment * 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:34 moritzm: installing libtasn1-6 security updates * 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) * 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:22 moritzm: installing glibc security updates * 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] * 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) * 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] * 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad * 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) * 11:50 jforrester@deploy1003: jforrester: Continuing with deployment * 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad * 11:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] * 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw * 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw * 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) * 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment * 10:44 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # [[phab:T439805|T439805]] * 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) * 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 * 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates * 08:05 moritzm: installing pcre2/openssl security updates * 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o) * 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) * 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:16 kharlan@deploy1003: kharlan: Continuing with deployment * 07:15 kharlan@deploy1003: kharlan: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] * 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:54 kharlan@deploy1003: kharlan: Continuing with deployment * 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:53 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet * 06:33 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] * 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) * 06:23 kharlan@deploy1003: kharlan: Continuing with deployment * 06:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:18 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] == 2026-09-30 == * 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 21:22 sbassett: Deployed updated mitigation for [[phab:T435455|T435455]] * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:52 arlolra@deploy1003: arlolra: Continuing with deployment * 20:52 arlolra@deploy1003: arlolra: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:47 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] * 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment * 20:31 mstyles@deploy1003: mstyles, catrope: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] * 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage * 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod * 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:22 brennen@deploy1003: Rolling back deployment * 18:10 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): no current blockers, rolling to group1 * 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:15 bking@dns2004: END - running authdns-update * 17:13 bking@dns2004: START - running authdns-update * 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet * 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet * 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet * 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:54 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet * 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:49 zabe@deploy1003: zabe: Continuing with deployment * 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:48 zabe@deploy1003: zabe: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet * 14:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] * 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet * 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet * 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet * 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) * 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:30 jforrester@deploy1003: jforrester: Continuing with deployment * 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:29 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) * 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: jforrester: Continuing with deployment * 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:16 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] * 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] (duration: 11m 33s) * 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 Amir1: mwscript-k8s --comment="[[phab:T358266|T358266]]" --local_dblist=<(expanddblist all {{!}} grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing ([[phab:T358266|T358266]]) * 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary [[phab:T437937|T437937]] * 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment * 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] * 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) * 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment * 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet * 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet * 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet * 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment * 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:25 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet * 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] * 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet * 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet * 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment * 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:10 samtar@deploy1003: migr, samtar: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] * 13:05 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] (duration: 10m 41s) * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet * 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet * 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) * 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - [[phab:T439667|T439667]] * 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet * 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:26 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] * 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet * 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet * 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet * 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 11:07 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] (duration: 11m 12s) * 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment * 10:59 cgoubert@deploy1003: cgoubert: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:56 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet * 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - [[phab:T438454|T438454]] * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:08 btullis@dns1004: END - running authdns-update * 09:06 btullis@dns1004: START - running authdns-update * 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' * 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' * 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad * 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad * 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca * 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw * 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw * 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:46 elukey@dns1004: END - running authdns-update * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:44 elukey@dns1004: START - running authdns-update * 07:35 moritzm: installing pcre2/openssl security updates * 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] (duration: 11m 27s) * 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:09 dcausse@deploy1003: dcausse: Continuing with deployment * 07:07 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] * 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet * 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet * 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-29 == * 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 * 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 * 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 * 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox * 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 * 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 * 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - [[phab:T427353|T427353]] [[phab:T395938|T395938]] * 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) * 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment * 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] * 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) * 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 * 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 * 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:43 cklimas@deploy1003: cklimas: Continuing with deployment * 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:38 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:33 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] * 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox * 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 * 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) * 20:19 sbassett@deploy1003: sbassett: Rolling back deployment * 20:12 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm * 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:08 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet * 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 18:06 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]) - no current blockers, rolling to group0 * 18:05 sandraebele: Deployed Refinery as part of weekly deployment train * 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) * 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] * 17:53 sandraebele: Deploying Refinery as part of weekly deployment train * 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) * 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] * 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie * 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train * 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 * 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` [[phab:T436186|T436186]] * 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # [[phab:T418194|T418194]] * 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply * 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply * 15:59 jhathaway: updating puppet stdlib, should be a noop * 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 14:21 moritzm: installing rsyslog security updates * 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet * 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet * 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw * 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:05 krinkle@deploy1003: krinkle: Continuing with deployment * 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox * 14:00 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 13:56 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] * 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] (duration: 00m 09s) * 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) * 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] * 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:25 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) * 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:20 samtar@deploy1003: samtar: Continuing with deployment * 13:19 samtar@deploy1003: samtar: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:14 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] * 13:10 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) * 13:05 samtar@deploy1003: samtar: Continuing with deployment * 13:04 samtar@deploy1003: samtar: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:00 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] * 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging * 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) * 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment * 12:44 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] * 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — [[phab:T432444|T432444]] * 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 * 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for [[phab:T417800|T417800]] (duration: 04m 14s) * 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for [[phab:T417800|T417800]] * 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox * 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet * 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # [[phab:T418194|T418194]] * 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) * 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment * 08:52 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] * 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad * 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad * 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad * 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:10 moritzm: installing PHP 8.4 security updates * 07:49 matthiasmullie: UTC morning backport window done * 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) * 07:44 mlitn@deploy1003: mlitn: Continuing with deployment * 07:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:39 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] * 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) * 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 07:30 kharlan@deploy1003: kharlan: Continuing with deployment * 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 07:27 kharlan@deploy1003: kharlan: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:26 kart_: Updated cxserver to 2026-09-25-114231-production * 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 07:23 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] * 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org * 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) * 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org * 07:11 mlitn@deploy1003: mlitn: Continuing with deployment * 07:09 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now * 07:04 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] * 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply * 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply * 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm * 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm * 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm * 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm * 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm * 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure * 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 * 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033 * 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] (duration: 36m 03s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie == 2026-09-28 == * 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:35 zabe: run script to fix fr_metadata drifts # [[phab:T420341|T420341]] * 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 21:47 sbassett: Deployed security mitigation for [[phab:T435455|T435455]] * 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync * 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync * 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - [[phab:T439201|T439201]] * 21:13 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) * 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment * 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 cjming@deploy1003: volker-e, cjming: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:00 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] * 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) * 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia [[phab:T438375|T438375]] * 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 20:46 kemayo@deploy1003: kemayo: Continuing with deployment * 20:45 kemayo@deploy1003: kemayo: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] * 20:32 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) * 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie * 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment * 20:26 cjming@deploy1003: ssastry, cjming: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:22 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] * 20:16 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) * 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment * 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.* * 20:07 aude@deploy1003: aude, volker-e: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:03 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie * 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts * 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s) * 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... * 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts * 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s) * 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie * 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie * 16:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) * 16:15 zabe@deploy1003: zabe: Continuing with deployment * 16:10 zabe@deploy1003: zabe: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:06 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] * 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] (duration: 19m 54s) * 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # [[phab:T439319|T439319]] * 15:45 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw * 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw * 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 14:55 Lucas_WMDE: UTC afternoon backport+config window done * 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) * 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment * 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) * 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] * 13:58 vgutierrez: repool cp7001 - [[phab:T439361|T439361]] * 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) * 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment * 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - [[phab:T432444|T432444]] * 13:47 bking@dns2004: END - running authdns-update * 13:44 bking@dns2004: START - running authdns-update * 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] * 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw * 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) * 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 vgutierrez: depool cp7001 for some purged tests - [[phab:T439361|T439361]] * 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw * 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw * 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] * 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) * 13:11 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] * 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet * 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:28 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) * 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:22 zabe@deploy1003: zabe: Continuing with deployment * 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet * 11:21 zabe@deploy1003: zabe: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] * 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 10:25 moritzm: installing chromium security updates * 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet * 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet * 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor * 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:16 moritzm: installing Exim security updates * 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:59 matthiasmullie: UTC morning backport window complete * 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) * 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment * 07:49 mlitn@deploy1003: migr, mlitn: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:44 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] * 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] (duration: 32m 06s) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:27 mlitn@deploy1003: mlitn: Continuing with deployment * 07:26 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi * 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] * 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-27 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-26 == * 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) * 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 * 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] (duration: 10m 53s) * 08:02 oblivian@deploy1003: oblivian: Continuing with deployment * 08:00 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:56 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] * 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] (duration: 11m 01s) * 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 03:30 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-25 == * 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values (duration: 00m 11s) * 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values * 17:59 cdanis@dns1004: END - running authdns-update * 17:57 cdanis@dns1004: START - running authdns-update * 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie * 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet * 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - [[phab:T343000|T343000]] * 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* * 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # [[phab:T438750|T438750]] * 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet * 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active * 13:24 Emperor: repool sessionstore in codfw * 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet * 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox * 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet * 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet * 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # [[phab:T438366|T438366]] * 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet * 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - [[phab:T439105|T439105]] * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet * 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet * 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox * 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet * 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.* * 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.* * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet == 2026-09-24 == * 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate * {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on t}} * 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment * 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment}} * {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on te}} * 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.* * 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet * 21:45 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s) * 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:40 catrope@deploy1003: catrope: Continuing with deployment * 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet * 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet * 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:33 catrope@deploy1003: catrope: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:28 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] * 21:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] (duration: 19m 27s) * 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment * 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi * 21:08 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] * 21:04 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s) * 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.* * 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.* * 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment * 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:53 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] * 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:49 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s) * 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:44 catrope@deploy1003: catrope: Continuing with deployment * 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:43 catrope@deploy1003: catrope: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet * 20:39 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s) * 20:14 kemayo@deploy1003: kemayo: Continuing with deployment * 20:12 kemayo@deploy1003: kemayo: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 20:08 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] * 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance * 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro * 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet` * 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet * 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness * 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance * 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly * 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance * 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw * 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance * 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters * 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian{{!}}tegola-vector-tiles) * 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s) * 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment * 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] * 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance * 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance * 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance * 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance * 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw * 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance * 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance * 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance * 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance * 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability * 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s) * 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.* * 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw * 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.* * 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift * 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master{{!}}device-analytics{{!}}echostore{{!}}helm-charts{{!}}k8s-ingress-wikikube-ro{{!}}linkrecommendation{{!}}mathoid{{!}}restbase{{!}}restbase-async{{!}}rest-gateway-ro{{!}}mobileapps{{!}}mwdebug.*{{!}}push-notifications{{!}}recommendation-api{{!}}releases{{!}}wikifeeds * 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid{{!}}zotero * 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium{{!}}schema{{!}}termbox{{!}}proton{{!}}cxserver * 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor * 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.* * 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.* * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro * 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions * 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s) * 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo * 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet * 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie * 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.* * 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes * 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet * 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] * 15:35 vgutierrez@dns1004: END - running authdns-update * 15:33 vgutierrez@dns1004: START - running authdns-update * 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s) * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet * 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie * 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox * 15:06 vgutierrez@dns1004: END - running authdns-update * 15:04 vgutierrez@dns1004: START - running authdns-update * 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update * 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s) * 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.* * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.* * 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:51 dkertesz: repooling cp7001{{!}}7011 after successful testing ([[phab:T343000|T343000]]) * 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 14:47 kharlan@deploy1003: kharlan: Continuing with deployment * 14:46 kharlan@deploy1003: kharlan: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.* * 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.* * 14:32 dkertesz: depooling cp7001{{!}}7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000) * 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof * 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:17 moritzm: installing Bird security updates * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie * 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.* * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s) * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie * 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] * 13:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s) * 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:35 moritzm: installing nghttp2 security updates * 13:35 awight@deploy1003: awight: Continuing with deployment * 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:33 awight@deploy1003: awight: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s) * 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment * 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki * 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.* * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:13 hnowlan: restarted restbase on restbase2029 * 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:50 hnowlan: deleting stuck mw-web pods in eqiad * 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) * 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision * 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] * 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) * 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision * 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] * 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) * 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw * 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:03 moritzm: installing apr-util security updates * 09:46 moritzm: installing bind9 security updates (client-side tools/libs only) * 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet * 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 * 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 * 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet * 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet * 09:01 XioNoX: asw1-b13-drmrs> request system reboot - [[phab:T437984|T437984]] * 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner * 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 * 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:58 moritzm: installing node-lodash security updates * 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 * 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision * 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 * 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 * 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade * 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade * 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet * 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:22 XioNoX: asw1-b12-drmrs> request system reboot - [[phab:T437984|T437984]] * 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 * 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 * 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner * 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade * 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade * 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production ([[phab:T437213|T437213]]) * 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply * 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production * 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply * 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw * 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw * 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green * 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here * 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected ([[phab:T439010|T439010]]) * 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet * 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected ([[phab:T439010|T439010]]) * 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected ([[phab:T439010|T439010]]) * 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts * 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though * 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) * 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state ([[phab:T439010|T439010]]) * 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* * 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-09-23 == * 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:56 brett: Switching acme-chief primary from codfw to eqiad - [[phab:T439010|T439010]] * 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones * 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia ([[phab:T438293|T438293]]) * 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. * 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 121m 40s) * 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next * 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next * 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected ([[phab:T439010|T439010]]) * 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see * 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected ([[phab:T439010|T439010]]) * 22:24 ryankemper: [Cirrus] s/expected/expect * 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel * 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) * 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status * 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 51m 29s) * 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 21:18 Emperor: ceph mgr fail on apus-be2005 * 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003 * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing * 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% * 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster * 20:49 swfrench@dns1004: END - running authdns-update * 20:46 swfrench@dns1004: START - running authdns-update * 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) * 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 20:06 dzahn@dns1004: END - running authdns-update * 20:03 dzahn@dns1004: START - running authdns-update * 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie * 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 19:33 volans: rebooting arclamp2001.codfw.wmnet * 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on * 19:17 taavi@dns1004: END - running authdns-update * 19:14 taavi@dns1004: START - running authdns-update * 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* * 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on * 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors * 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors * 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org * 18:45 sukhe@dns1004: END - running authdns-update * 18:43 sukhe@dns1004: START - running authdns-update * 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org * 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org * 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie * 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors * 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors * 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:01 cdanis@dns1005: END - running authdns-update * 17:58 cdanis@dns1005: START - running authdns-update * 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:54 taavi@dns1004: END - running authdns-update * 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:51 taavi@dns1004: START - running authdns-update * 17:46 taavi@dns1004: END - running authdns-update * 17:43 taavi@dns1004: START - running authdns-update * 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - [[phab:T439010|T439010]] * 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie * 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 * 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 * 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie * 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie * 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:54 moritzm: installing cjose security updates * 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* * 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) * 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:46 dancy@deploy1003: Started scap sync-world: testing * 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie * 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' * 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 [[phab:T438961|T438961]] * 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 * 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors * 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ [[phab:T438961|T438961]] * 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d [[phab:T438961|T438961]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before [[phab:T438786|T438786]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox * 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 [[phab:T438961|T438961]] * 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie * 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 * 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts * 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) * 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance * 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance * 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) * 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie * 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* * 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] * 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie * 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:56 awight: manually run puppet agent * 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:45 awight: add seanleong-wmde to deployment-prep * 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad * 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 12:22 kharlan@deploy1003: kharlan: Continuing with deployment * 12:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:01 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] * 11:51 Dreamy_Jazz: Deployed patch for [[phab:T438729|T438729]] * 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - [[phab:T438896|T438896]] * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web [[phab:T438896|T438896]] (duration: 04m 20s) * 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web [[phab:T438896|T438896]] * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:13 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] (duration: 12m 52s) * 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts * 08:04 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] * 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:59 moritzm: installing giflib security updates * 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:29 moritzm: installing python-idna security updates * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance * 00:46 ryankemper: [WDQS] [[phab:T435443|T435443]] Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture * 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance == 2026-09-22 == * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] (duration: 13m 43s) * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 21:53 rzl@deploy1003: rzl: Continuing with deployment * 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] * 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) * 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) * 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet * 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) * 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] * 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) * 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) * 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] * 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:40 sbisson@deploy1003: sbisson: Continuing with deployment * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 sbisson@deploy1003: sbisson: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:35 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] * 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] (duration: 13m 35s) * 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment * 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] * 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] (duration: 10m 29s) * 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.* * 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:11 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] * 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie * 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie * 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0 * 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts * 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s) * 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts * 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s) * 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0 * 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] (duration: 10m 09s) * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030 * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030 * 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030 * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] * 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030 * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029 * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029 * 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029 * 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet * 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:24 dzahn@dns1004: END - running authdns-update * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 17:22 dzahn@dns1004: START - running authdns-update * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed * 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] (duration: 13m 13s) * 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance * 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance * 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing * 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] * 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing * 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance * 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance * 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues * 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 15:50 moritzm: installing libhtml-parser-perl security updates * 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance * 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox * 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120 * 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120 * 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120 * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts * 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s) * 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s) * 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN [[phab:T436571|T436571]] * 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120 * 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002 * 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002 * 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 12:44 marostegui: Stop mariadb on db2250:s5 [[phab:T437411|T437411]] [[phab:T437279|T437279]] * 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] (duration: 33m 01s) * 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:08 kharlan@deploy1003: kharlan: Continuing with deployment * 12:06 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host (duration: 00m 54s) * 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host * 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host (duration: 01m 01s) * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host * 11:46 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] * 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] (duration: 13m 46s) * 11:34 kharlan@deploy1003: kharlan: Continuing with deployment * 11:33 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] * 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] (duration: 33m 09s) * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 11:13 kharlan@deploy1003: kharlan: Continuing with deployment * 11:12 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922 * 10:51 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] * 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie * 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 * 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 * 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 * 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 * 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 * 10:06 moritzm: installing libcap2 security updates * 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 * 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 * 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 * 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 * 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 * 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 * 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 * 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 * 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 * 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003 * 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 * 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues * 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues * 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie * 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - [[phab:T438828|T438828]] * 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues * 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues * 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations * 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet * 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:26 btullis@dns1004: END - running authdns-update * 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:23 btullis@dns1004: START - running authdns-update * 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet * 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 * 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 * 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3 * 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320 * 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320 * 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3 * 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1 * 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1 * 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:32 moritzm: installig zip security updates * 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:29 XioNoX: asw1-bw27-esams> request system reboot - [[phab:T437984|T437984]] * 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27 * 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27 * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:18 moritzm: installing gst-plugins-base1.0 security updates * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot * 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot * 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:23 awight: UTC morning deployment window complete * 07:22 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] (duration: 17m 46s) * 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment * 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] * 07:02 moritzm: installing pyasn1 security updates * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] (duration: 35m 52s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-21 == * 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:18 maryum: Deployed security fix for [[phab:T437708|T437708]] * 20:35 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] (duration: 15m 56s) * 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment * 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:19 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired * 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.* * 18:30 Emperor: repool eqiad sessionstore [[phab:T437915|T437915]] * 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired * 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie * 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm * 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie * 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.* * 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006 * 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie * 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie * 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] (duration: 12m 40s) * 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie * 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:36 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply * 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply * 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] * 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.* * 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie * 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.* * 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie * 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply * 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply * 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie * 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:13 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] (duration: 08m 09s) * 14:08 zabe@deploy1003: zabe: Continuing with deployment * 14:08 zabe@deploy1003: zabe: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie * 14:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 14:01 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] (duration: 12m 30s) * 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment * 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # [[phab:T438716|T438716]] * 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] (duration: 11m 40s) * 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment * 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] * 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] (duration: 14m 19s) * 13:15 mfossati@deploy1003: mfossati: Continuing with deployment * 13:14 mfossati@deploy1003: mfossati: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet * 13:07 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] * 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org * 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet * 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX) * 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet * 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet * 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet * 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # [[phab:T438499|T438499]] * 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet * 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie * 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:35 moritzm: installing chromium security updates * 09:22 tappof: bump space for prometheus k8s-dse in eqiad * 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s) * 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie * 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] * 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision * 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - [[phab:T438291|T438291]] * 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - [[phab:T438291|T438291]] * 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - [[phab:T438291|T438291]] * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw * 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet * 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003 * 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 [[phab:T435314|T435314]] * 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org == 2026-09-20 == * 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply == 2026-09-19 == * 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually ([[phab:T438591|T438591]], see [[phab:T438591|T438591]]#12341750 for what I did exactly) * 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ni4qr5yf0qemlz24rb3vex5hi6mdvqt 2466281 2466280 2026-10-11T02:00:22Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2466281 wikitext text/x-wiki == 2026-10-11 == * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-10 == * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-09 == * 18:54 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) * 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment * 18:48 cscott@deploy1003: cscott, ssastry: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:44 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] * 17:38 Daimona: Run database query from [[phab:T440557|T440557]]#12411029 in x1.wikishared * 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 * 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet * 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606{{!}}T440606]]' * 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet * 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet * 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet * 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet * 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location * 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet * 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet * 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet * 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet * 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test * 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test * 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test * 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm * 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A * 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch * 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump * 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm * 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet * 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet * 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet * 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) * 01:25 kemayo@deploy1003: kemayo: Continuing with deployment * 01:24 kemayo@deploy1003: kemayo: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 01:20 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] * 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr [[phab:T440068|T440068]] - downloads 120x times faster == 2026-10-08 == * 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning * 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) * 23:16 ejegg@deploy1003: ejegg: Continuing with deployment * 23:11 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:51 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] * 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) * 22:25 kemayo@deploy1003: kemayo: Continuing with deployment * 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 22:20 kemayo@deploy1003: kemayo: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 22:00 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] * 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) * 21:37 ejegg@deploy1003: ejegg: Rolling back deployment * 21:27 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad * 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:07 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] * 20:48 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) * 20:43 cjming@deploy1003: cjming: Continuing with deployment * 20:42 cjming@deploy1003: cjming: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:38 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] * 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] (duration: 09m 41s) * 20:27 mlitn@deploy1003: mlitn: Continuing with deployment * 20:27 mlitn@deploy1003: mlitn: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:23 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] * 20:21 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) * 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment * 20:10 cjming@deploy1003: milazg, cjming: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] * 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) * 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update * 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet * 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet * 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet * 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes * 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts * 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s) * 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad * 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change * 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie * 16:07 swfrench@dns1004: END - running authdns-update * 16:04 swfrench@dns1004: START - running authdns-update * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) * 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet * 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet * 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet * 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning * 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet * 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet * 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes * 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:54 cmooney@dns3003: END - running authdns-update * 14:51 cmooney@dns3003: START - running authdns-update * 14:50 cgoubert@dns2004: END - running authdns-update * 14:48 cgoubert@dns2004: START - running authdns-update * 14:48 moritzm: installing pyasn1 security updates * 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet * 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] (duration: 36m 09s) * 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet * 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 14:23 moritzm: installing perl security updates * 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet * 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 14:03 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:47 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) * 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet * 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment * 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet * 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet * 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet * 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet * 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet * 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # [[phab:T440023|T440023]] [[phab:T439902|T439902]] * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet * 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet * 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet * 12:30 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) * 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet * 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet * 12:25 zabe@deploy1003: zabe: Continuing with deployment * 12:24 zabe@deploy1003: zabe: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet * 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] * 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet * 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet * 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet * 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet * 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet * 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] (duration: 04m 15s) * 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet * 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet * 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet * 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] * 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:10 topranks: cr1-codfw switch back to master routing engine [[phab:T426197|T426197]] * 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet * 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:36 topranks: cr1-codfw switch to backup routing engine [[phab:T426197|T426197]] * 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host * 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet * 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet * 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet * 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover * 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet * 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet * 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host * 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade [[phab:T417871|T417871]] * 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet * 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet * 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet * 08:59 moritzm: installing Wireshark security updates * 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json * 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json * 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - [[phab:T436922|T436922]] * 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json * 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 [[phab:T436922|T436922]] * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet * 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet * 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:42 moritzm: installing python3.13 security updates * 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet * 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:01 moritzm: installing mbedtls security updates * 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet * 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet * 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) * 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet * 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment * 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet * 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet * 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet * 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet * 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups * 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet * 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet * 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet * 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet * 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old ([[phab:T440358|T440358]]) * 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet * 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet * 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet * 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet * 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet * 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet * 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet * 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet * 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet * 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet * 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet * 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet * 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet * 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet * 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet * 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet * 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet * 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet * 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet * 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet * 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet * 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet * 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet * 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet * 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet * 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet * 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet * 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet * 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet * 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet * 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet * 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet * 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font ([[phab:T393763|T393763]]) == 2026-10-07 == * 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm * 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet * 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) * 20:23 kemayo@deploy1003: kemayo: Continuing with deployment * 20:21 kemayo@deploy1003: kemayo: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] * 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply * 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply * 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts * 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s) * 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet * 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] (duration: 10m 40s) * 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] * 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet * 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet * 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for [[phab:T417800|T417800]] (duration: 05m 23s) * 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:17 blake@deploy1003: Started scap sync-world: add restart policies for [[phab:T417800|T417800]] * 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet * 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet * 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) * 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet * 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) * 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet * 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) * 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade * 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade * 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet * 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting * 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade * 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade * 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade * 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade * 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade * 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) * 14:48 jforrester@deploy1003: jforrester: Continuing with deployment * 14:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams * 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams * 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs * 14:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] * 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs * 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin * 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo * 14:34 jforrester@deploy1003: jforrester: Continuing with deployment * 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo * 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:27 moritzm: installing PHP 8.2 security updates * 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru * 14:26 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet * 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet * 14:22 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] * 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 14:19 moritzm: installing postgresql-15 security updates * 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - [[phab:T439840|T439840]] * 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet * 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet * 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment * 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:54 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet * 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 13:44 papaul: start maintenance on cr2-codfw * 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) * 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet * 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] * 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) * 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:13 moritzm: installing mbedtls security updates * 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) * 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet * 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply * 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet * 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet * 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet * 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet * 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet * 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet * 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet * 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet * 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 andre: Rolling back again wmf.23 from group1 to group0 - [[phab:T438219|T438219]] * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet * 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet * 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for [[phab:T417800|T417800]] (duration: 07m 32s) * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for [[phab:T417800|T417800]] * 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet * 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet * 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet * 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet * 09:54 duesen: just a test for SAL * 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:50 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) * 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet * 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:36 daniel@deploy1003: daniel: Continuing with deployment * 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet * 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet * 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:26 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet * 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet * 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet * 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet * 09:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:13 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] * 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) * 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] * 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) * 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] * 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet * 08:20 aklapper@deploy1003: Rolling back deployment * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet * 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet * 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet * 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet * 08:02 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) * 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment * 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet * 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia [[phab:T392465|T392465]] * 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:25 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] * 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet * 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet * 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet * 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 moritzm: installing bird security updates * 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet * 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet * 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet * 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet * 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet * 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet * 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet * 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:10 kart_: Updated Apertium to 2026-09-30-051826-production ([[phab:T437213|T437213]]) * 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply * 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply * 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply * 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply * 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host [[phab:T437279|T437279]] * 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning * 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host * 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply * 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-06 == * 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) * 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:00 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] * 20:15 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 * 20:10 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt * 20:03 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] * 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test * 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet * 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet * 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test * 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet * 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet * 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test * 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet * 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet * 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet * 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet * 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet * 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet * 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet * 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet * 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet * 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:39 bking@dns2004: END - running authdns-update * 16:37 bking@dns2004: START - running authdns-update * 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet * 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet * 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet * 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet * 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:26 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) * 15:21 daniel@deploy1003: daniel: Continuing with deployment * 15:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing python3.13 security updates * 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 15:11 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] * 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw * 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] (duration: 00m 42s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] * 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment * 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] (duration: 00m 51s) * 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] * 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment * 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:57 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet * 14:48 moritzm: restarting pybal on lvs1020 [[phab:T439730|T439730]] * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw * 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet * 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) * 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment * 14:33 moritzm: restarting pybal on lvs2012 [[phab:T439730|T439730]] * 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet * 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet * 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) * 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet * 14:23 moritzm: restarting pybal on lvs2014 [[phab:T439730|T439730]] * 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb * 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb * 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet * 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet * 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb * 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet * 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet * 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding chec}} * 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment * 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin * 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet * {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a s}} * 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet * 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding check}} * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment * 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] * 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 moritzm: installing php8.2 security updates * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 214m 10s) * 13:31 slyngshede@dns1004: END - running authdns-update * 13:29 slyngshede@dns1004: START - running authdns-update * 13:25 moritzm: installing libssh2 security updates * 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 13:14 moritzm: installing glibc security updates * 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 11:33 moritzm: installing chromium security updates * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply * 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply * 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 * 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back * 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor ([[phab:T406724|T406724]]) * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading * 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 [[phab:T436082|T436082]] * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 moritzm: installing curl security updates * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing * 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m {{!}} tee gc_run_06102026` - [[phab:T437297|T437297]] * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 * 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 00m 19s) * 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 * 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back * 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 * 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing * 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing * 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning * 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning * 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning * 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 * 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back * 08:00 matthiasmullie: UTC morning backport window done * 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) * 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:50 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] * 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) * 07:38 dcausse@deploy1003: dcausse: Continuing with deployment * 07:38 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 07:31 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] * 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 * 07:21 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) * 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back * 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment * 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] * 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 * 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 * 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning * 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 * 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) * 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] (duration: 27m 17s) * 03:28 mwpresync@deploy1003: Rolling back deployment * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-05 == * 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) * 23:18 derenrich@deploy1003: derenrich: Continuing with deployment * 23:16 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:12 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet * 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439719|T439719]] * 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439974|T439974]] * 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:51 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439932|T439932]] * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams * 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs * 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs * 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb) * 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # [[phab:T439577|T439577]] * 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) * 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes * 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply * 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply * 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru * 16:32 dcausse: closing the "Distributed search queries experiment" window * 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) * 16:25 dcausse@deploy1003: dcausse: Continuing with deployment * 16:23 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox * 16:19 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] * 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad * 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) * 15:59 dcausse@deploy1003: dcausse: Continuing with deployment * 15:58 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:54 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] * 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad * 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] (duration: 25m 12s) * 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm * 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet * 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox * 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:27 dcausse@deploy1003: dcausse: Continuing with deployment * 15:26 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet * 15:22 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm * 15:16 elukey@dns1004: END - running authdns-update * 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] (duration: 10m 11s) * 15:14 elukey@dns1004: START - running authdns-update * 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet * 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:10 dcausse@deploy1003: dcausse: Continuing with deployment * 15:09 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:05 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet * 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] (duration: 10m 12s) * 14:53 dcausse@deploy1003: dcausse: Continuing with deployment * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:52 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet * 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) * 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] (duration: 10m 04s) * 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet * 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet * 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 dcausse@deploy1003: dcausse: Continuing with deployment * 14:32 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:28 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] * 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) * 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:15 dcausse@deploy1003: dcausse: Continuing with deployment * 14:13 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:09 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439968|T439968]] * 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet * 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet * 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] (duration: 09m 59s) * 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service * 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:37 zabe@deploy1003: zabe: Continuing with deployment * 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:36 zabe@deploy1003: zabe: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] * 13:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) * 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet * 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet * 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet * 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:21 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] * 13:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet * 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet * 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment * 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet * 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet * 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet * 13:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet * 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet * 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet * 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie * 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) * 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:31 moritzm: installing sqlite3 security updates * 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment * 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:16 moritzm: installing mesa security updates * 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org * 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org * 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org * 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org * 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005 * 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm * 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet * 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] * 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet * 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet * 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet * 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet * 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet * 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:46 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] (duration: 04m 44s) * 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:42 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] * 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing * 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests * 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing * 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests * 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm * 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests * 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests * 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing * 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing * 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet * 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet * 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing * 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm * 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests * 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests * 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet * 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet * 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie * 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for [[phab:T417800|T417800]] (duration: 09m 40s) * 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for [[phab:T417800|T417800]] * 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet * 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet * 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:22 volans: upgrading cloudcumin2001 in-place to trixie [[phab:T436667|T436667]] * 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm * 07:49 dpogorzelski@dns1004: END - running authdns-update * 07:46 dpogorzelski@dns1004: START - running authdns-update * 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-04 == * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-03 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-02 == * 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` [[phab:T440033|T440033]] * 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - [[phab:T427353|T427353]] * 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie * 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet * 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet * 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet * 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet * 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet * 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet * 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet * 17:44 dzahn@dns1004: END - running authdns-update * 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet * 17:41 dzahn@dns1004: START - running authdns-update * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet * 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet * 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 * 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 * 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox * 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 * 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet * 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet * 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes * 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) * 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes * 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet * 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm * 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet * 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox * 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm * 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox * 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts * 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm * 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox * 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts * 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-10-01 == * 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.* * 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) * 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] * 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — [[phab:T432444|T432444]] * 21:24 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) * 21:19 egardner@deploy1003: egardner: Continuing with deployment * 21:17 egardner@deploy1003: egardner: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] * 21:06 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) * 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment * 20:59 brennen@deploy1003: brennen, srapoj: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] * 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - [[phab:T432444|T432444]] * 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) * 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:37 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] * 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) * 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment * 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:18 cjming@deploy1003: cjming, volker-e: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can * 20:14 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] * 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two * 19:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] (duration: 09m 40s) * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:46 zabe@deploy1003: zabe: Continuing with deployment * 19:46 zabe@deploy1003: zabe: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:42 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] * 19:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) * 19:36 zabe@deploy1003: zabe: Continuing with deployment * 19:36 zabe@deploy1003: zabe: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] * 19:19 jhathaway: puppet re-enabled * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 * 18:25 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): logs kind of a mess, but no current blockers, rolling to all wikis * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox * 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T439100|T439100]] (duration: 33m 25s) * 17:16 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T439100|T439100]] * 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:32 moritzm: installing socat security updates * 15:29 dpogorzelski@dns1004: END - running authdns-update * 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:25 dpogorzelski@dns1004: START - running authdns-update * 15:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] (duration: 10m 12s) * 15:14 zabe@deploy1003: zabe: Continuing with deployment * 15:14 zabe@deploy1003: zabe: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing postgresql security updates * 15:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] * 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.* * 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.* * 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release * 14:48 moritzm: installing busybox bugfix updates from trixie point release * 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) * 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment * 14:42 moritzm: installing zsh bugfix update from trixie point release * 14:41 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] * 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release * 14:31 dpogorzelski@dns1004: END - running authdns-update * 14:29 dpogorzelski@dns1004: START - running authdns-update * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 14:04 awight: EU afternoon backport complete * 14:03 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) * 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:58 awight@deploy1003: lilients, awight: Continuing with deployment * 13:57 awight@deploy1003: lilients, awight: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:53 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:41 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) * 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:36 awight@deploy1003: awight, migr: Continuing with deployment * 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 awight@deploy1003: awight, migr: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve * 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:18 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] * 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment * 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:34 moritzm: installing libtasn1-6 security updates * 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) * 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:22 moritzm: installing glibc security updates * 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] * 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) * 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] * 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad * 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) * 11:50 jforrester@deploy1003: jforrester: Continuing with deployment * 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad * 11:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] * 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw * 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw * 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) * 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment * 10:44 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # [[phab:T439805|T439805]] * 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) * 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 * 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates * 08:05 moritzm: installing pcre2/openssl security updates * 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o) * 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) * 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:16 kharlan@deploy1003: kharlan: Continuing with deployment * 07:15 kharlan@deploy1003: kharlan: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] * 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:54 kharlan@deploy1003: kharlan: Continuing with deployment * 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:53 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet * 06:33 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] * 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) * 06:23 kharlan@deploy1003: kharlan: Continuing with deployment * 06:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:18 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] == 2026-09-30 == * 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 21:22 sbassett: Deployed updated mitigation for [[phab:T435455|T435455]] * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:52 arlolra@deploy1003: arlolra: Continuing with deployment * 20:52 arlolra@deploy1003: arlolra: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:47 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] * 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment * 20:31 mstyles@deploy1003: mstyles, catrope: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] * 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage * 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod * 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:22 brennen@deploy1003: Rolling back deployment * 18:10 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): no current blockers, rolling to group1 * 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:15 bking@dns2004: END - running authdns-update * 17:13 bking@dns2004: START - running authdns-update * 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet * 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet * 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet * 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:54 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet * 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:49 zabe@deploy1003: zabe: Continuing with deployment * 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:48 zabe@deploy1003: zabe: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet * 14:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] * 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet * 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet * 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet * 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) * 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:30 jforrester@deploy1003: jforrester: Continuing with deployment * 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:29 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) * 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: jforrester: Continuing with deployment * 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:16 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] * 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] (duration: 11m 33s) * 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 Amir1: mwscript-k8s --comment="[[phab:T358266|T358266]]" --local_dblist=<(expanddblist all {{!}} grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing ([[phab:T358266|T358266]]) * 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary [[phab:T437937|T437937]] * 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment * 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] * 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) * 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment * 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet * 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet * 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet * 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment * 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:25 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet * 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] * 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet * 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet * 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment * 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:10 samtar@deploy1003: migr, samtar: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] * 13:05 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] (duration: 10m 41s) * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet * 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet * 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) * 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - [[phab:T439667|T439667]] * 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet * 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:26 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] * 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet * 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet * 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet * 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 11:07 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] (duration: 11m 12s) * 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment * 10:59 cgoubert@deploy1003: cgoubert: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:56 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet * 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - [[phab:T438454|T438454]] * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:08 btullis@dns1004: END - running authdns-update * 09:06 btullis@dns1004: START - running authdns-update * 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' * 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' * 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad * 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad * 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca * 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw * 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw * 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:46 elukey@dns1004: END - running authdns-update * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:44 elukey@dns1004: START - running authdns-update * 07:35 moritzm: installing pcre2/openssl security updates * 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] (duration: 11m 27s) * 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:09 dcausse@deploy1003: dcausse: Continuing with deployment * 07:07 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] * 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet * 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet * 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-29 == * 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 * 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 * 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 * 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox * 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 * 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 * 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - [[phab:T427353|T427353]] [[phab:T395938|T395938]] * 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) * 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment * 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] * 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) * 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 * 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 * 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:43 cklimas@deploy1003: cklimas: Continuing with deployment * 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:38 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:33 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] * 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox * 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 * 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) * 20:19 sbassett@deploy1003: sbassett: Rolling back deployment * 20:12 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm * 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:08 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet * 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 18:06 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]) - no current blockers, rolling to group0 * 18:05 sandraebele: Deployed Refinery as part of weekly deployment train * 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) * 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] * 17:53 sandraebele: Deploying Refinery as part of weekly deployment train * 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) * 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] * 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie * 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train * 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 * 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` [[phab:T436186|T436186]] * 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # [[phab:T418194|T418194]] * 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply * 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply * 15:59 jhathaway: updating puppet stdlib, should be a noop * 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 14:21 moritzm: installing rsyslog security updates * 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet * 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet * 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw * 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:05 krinkle@deploy1003: krinkle: Continuing with deployment * 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox * 14:00 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 13:56 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] * 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] (duration: 00m 09s) * 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) * 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] * 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:25 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) * 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:20 samtar@deploy1003: samtar: Continuing with deployment * 13:19 samtar@deploy1003: samtar: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:14 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] * 13:10 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) * 13:05 samtar@deploy1003: samtar: Continuing with deployment * 13:04 samtar@deploy1003: samtar: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:00 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] * 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging * 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) * 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment * 12:44 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] * 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — [[phab:T432444|T432444]] * 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 * 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for [[phab:T417800|T417800]] (duration: 04m 14s) * 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for [[phab:T417800|T417800]] * 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox * 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet * 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # [[phab:T418194|T418194]] * 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) * 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment * 08:52 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] * 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad * 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad * 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad * 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:10 moritzm: installing PHP 8.4 security updates * 07:49 matthiasmullie: UTC morning backport window done * 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) * 07:44 mlitn@deploy1003: mlitn: Continuing with deployment * 07:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:39 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] * 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) * 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 07:30 kharlan@deploy1003: kharlan: Continuing with deployment * 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 07:27 kharlan@deploy1003: kharlan: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:26 kart_: Updated cxserver to 2026-09-25-114231-production * 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 07:23 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] * 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org * 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) * 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org * 07:11 mlitn@deploy1003: mlitn: Continuing with deployment * 07:09 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now * 07:04 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] * 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply * 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply * 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm * 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm * 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm * 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm * 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm * 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure * 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 * 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033 * 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] (duration: 36m 03s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie == 2026-09-28 == * 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:35 zabe: run script to fix fr_metadata drifts # [[phab:T420341|T420341]] * 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 21:47 sbassett: Deployed security mitigation for [[phab:T435455|T435455]] * 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync * 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync * 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - [[phab:T439201|T439201]] * 21:13 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) * 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment * 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 cjming@deploy1003: volker-e, cjming: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:00 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] * 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) * 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia [[phab:T438375|T438375]] * 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 20:46 kemayo@deploy1003: kemayo: Continuing with deployment * 20:45 kemayo@deploy1003: kemayo: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] * 20:32 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) * 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie * 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment * 20:26 cjming@deploy1003: ssastry, cjming: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:22 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] * 20:16 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) * 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment * 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.* * 20:07 aude@deploy1003: aude, volker-e: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:03 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie * 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts * 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s) * 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... * 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts * 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s) * 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie * 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie * 16:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) * 16:15 zabe@deploy1003: zabe: Continuing with deployment * 16:10 zabe@deploy1003: zabe: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:06 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] * 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] (duration: 19m 54s) * 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # [[phab:T439319|T439319]] * 15:45 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw * 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw * 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 14:55 Lucas_WMDE: UTC afternoon backport+config window done * 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) * 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment * 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) * 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] * 13:58 vgutierrez: repool cp7001 - [[phab:T439361|T439361]] * 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) * 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment * 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - [[phab:T432444|T432444]] * 13:47 bking@dns2004: END - running authdns-update * 13:44 bking@dns2004: START - running authdns-update * 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] * 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw * 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) * 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 vgutierrez: depool cp7001 for some purged tests - [[phab:T439361|T439361]] * 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw * 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw * 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] * 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) * 13:11 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] * 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet * 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:28 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) * 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:22 zabe@deploy1003: zabe: Continuing with deployment * 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet * 11:21 zabe@deploy1003: zabe: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] * 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 10:25 moritzm: installing chromium security updates * 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet * 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet * 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor * 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:16 moritzm: installing Exim security updates * 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:59 matthiasmullie: UTC morning backport window complete * 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) * 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment * 07:49 mlitn@deploy1003: migr, mlitn: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:44 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] * 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] (duration: 32m 06s) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:27 mlitn@deploy1003: mlitn: Continuing with deployment * 07:26 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi * 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] * 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-27 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-26 == * 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) * 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 * 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] (duration: 10m 53s) * 08:02 oblivian@deploy1003: oblivian: Continuing with deployment * 08:00 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:56 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] * 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] (duration: 11m 01s) * 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 03:30 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-25 == * 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values (duration: 00m 11s) * 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values * 17:59 cdanis@dns1004: END - running authdns-update * 17:57 cdanis@dns1004: START - running authdns-update * 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie * 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet * 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - [[phab:T343000|T343000]] * 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* * 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # [[phab:T438750|T438750]] * 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet * 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active * 13:24 Emperor: repool sessionstore in codfw * 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet * 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox * 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet * 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet * 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # [[phab:T438366|T438366]] * 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet * 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - [[phab:T439105|T439105]] * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet * 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet * 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox * 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet * 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.* * 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.* * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet == 2026-09-24 == * 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate * {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on t}} * 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment * 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment}} * {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on te}} * 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.* * 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet * 21:45 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s) * 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:40 catrope@deploy1003: catrope: Continuing with deployment * 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet * 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet * 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:33 catrope@deploy1003: catrope: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:28 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] * 21:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] (duration: 19m 27s) * 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment * 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi * 21:08 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] * 21:04 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s) * 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.* * 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.* * 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment * 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:53 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] * 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:49 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s) * 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:44 catrope@deploy1003: catrope: Continuing with deployment * 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:43 catrope@deploy1003: catrope: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet * 20:39 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s) * 20:14 kemayo@deploy1003: kemayo: Continuing with deployment * 20:12 kemayo@deploy1003: kemayo: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 20:08 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] * 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance * 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro * 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet` * 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet * 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness * 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance * 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly * 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance * 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw * 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance * 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters * 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian{{!}}tegola-vector-tiles) * 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s) * 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment * 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] * 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance * 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance * 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance * 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance * 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw * 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance * 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance * 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance * 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance * 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability * 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s) * 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.* * 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw * 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.* * 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift * 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master{{!}}device-analytics{{!}}echostore{{!}}helm-charts{{!}}k8s-ingress-wikikube-ro{{!}}linkrecommendation{{!}}mathoid{{!}}restbase{{!}}restbase-async{{!}}rest-gateway-ro{{!}}mobileapps{{!}}mwdebug.*{{!}}push-notifications{{!}}recommendation-api{{!}}releases{{!}}wikifeeds * 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid{{!}}zotero * 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium{{!}}schema{{!}}termbox{{!}}proton{{!}}cxserver * 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor * 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.* * 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.* * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro * 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions * 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s) * 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo * 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet * 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie * 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.* * 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes * 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet * 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] * 15:35 vgutierrez@dns1004: END - running authdns-update * 15:33 vgutierrez@dns1004: START - running authdns-update * 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s) * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet * 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie * 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox * 15:06 vgutierrez@dns1004: END - running authdns-update * 15:04 vgutierrez@dns1004: START - running authdns-update * 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update * 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s) * 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.* * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.* * 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:51 dkertesz: repooling cp7001{{!}}7011 after successful testing ([[phab:T343000|T343000]]) * 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 14:47 kharlan@deploy1003: kharlan: Continuing with deployment * 14:46 kharlan@deploy1003: kharlan: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.* * 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.* * 14:32 dkertesz: depooling cp7001{{!}}7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000) * 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof * 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:17 moritzm: installing Bird security updates * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie * 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.* * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s) * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie * 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] * 13:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s) * 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:35 moritzm: installing nghttp2 security updates * 13:35 awight@deploy1003: awight: Continuing with deployment * 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:33 awight@deploy1003: awight: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s) * 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment * 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki * 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.* * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:13 hnowlan: restarted restbase on restbase2029 * 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:50 hnowlan: deleting stuck mw-web pods in eqiad * 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) * 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision * 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] * 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) * 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision * 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] * 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) * 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw * 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:03 moritzm: installing apr-util security updates * 09:46 moritzm: installing bind9 security updates (client-side tools/libs only) * 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet * 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 * 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 * 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet * 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet * 09:01 XioNoX: asw1-b13-drmrs> request system reboot - [[phab:T437984|T437984]] * 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner * 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 * 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:58 moritzm: installing node-lodash security updates * 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 * 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision * 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 * 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 * 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade * 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade * 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet * 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:22 XioNoX: asw1-b12-drmrs> request system reboot - [[phab:T437984|T437984]] * 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 * 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 * 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner * 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade * 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade * 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production ([[phab:T437213|T437213]]) * 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply * 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production * 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply * 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw * 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw * 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green * 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here * 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected ([[phab:T439010|T439010]]) * 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet * 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected ([[phab:T439010|T439010]]) * 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected ([[phab:T439010|T439010]]) * 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts * 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though * 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) * 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state ([[phab:T439010|T439010]]) * 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* * 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-09-23 == * 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:56 brett: Switching acme-chief primary from codfw to eqiad - [[phab:T439010|T439010]] * 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones * 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia ([[phab:T438293|T438293]]) * 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. * 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 121m 40s) * 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next * 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next * 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected ([[phab:T439010|T439010]]) * 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see * 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected ([[phab:T439010|T439010]]) * 22:24 ryankemper: [Cirrus] s/expected/expect * 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel * 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) * 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status * 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 51m 29s) * 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 21:18 Emperor: ceph mgr fail on apus-be2005 * 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003 * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing * 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% * 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster * 20:49 swfrench@dns1004: END - running authdns-update * 20:46 swfrench@dns1004: START - running authdns-update * 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) * 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 20:06 dzahn@dns1004: END - running authdns-update * 20:03 dzahn@dns1004: START - running authdns-update * 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie * 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 19:33 volans: rebooting arclamp2001.codfw.wmnet * 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on * 19:17 taavi@dns1004: END - running authdns-update * 19:14 taavi@dns1004: START - running authdns-update * 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* * 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on * 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors * 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors * 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org * 18:45 sukhe@dns1004: END - running authdns-update * 18:43 sukhe@dns1004: START - running authdns-update * 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org * 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org * 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie * 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors * 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors * 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:01 cdanis@dns1005: END - running authdns-update * 17:58 cdanis@dns1005: START - running authdns-update * 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:54 taavi@dns1004: END - running authdns-update * 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:51 taavi@dns1004: START - running authdns-update * 17:46 taavi@dns1004: END - running authdns-update * 17:43 taavi@dns1004: START - running authdns-update * 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - [[phab:T439010|T439010]] * 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie * 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 * 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 * 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie * 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie * 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:54 moritzm: installing cjose security updates * 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* * 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) * 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:46 dancy@deploy1003: Started scap sync-world: testing * 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie * 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' * 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 [[phab:T438961|T438961]] * 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 * 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors * 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ [[phab:T438961|T438961]] * 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d [[phab:T438961|T438961]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before [[phab:T438786|T438786]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox * 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 [[phab:T438961|T438961]] * 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie * 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 * 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts * 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) * 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance * 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance * 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) * 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie * 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* * 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] * 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie * 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:56 awight: manually run puppet agent * 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:45 awight: add seanleong-wmde to deployment-prep * 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad * 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 12:22 kharlan@deploy1003: kharlan: Continuing with deployment * 12:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:01 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] * 11:51 Dreamy_Jazz: Deployed patch for [[phab:T438729|T438729]] * 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - [[phab:T438896|T438896]] * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web [[phab:T438896|T438896]] (duration: 04m 20s) * 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web [[phab:T438896|T438896]] * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:13 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] (duration: 12m 52s) * 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts * 08:04 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] * 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:59 moritzm: installing giflib security updates * 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:29 moritzm: installing python-idna security updates * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance * 00:46 ryankemper: [WDQS] [[phab:T435443|T435443]] Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture * 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance == 2026-09-22 == * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] (duration: 13m 43s) * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 21:53 rzl@deploy1003: rzl: Continuing with deployment * 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] * 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) * 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) * 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet * 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) * 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] * 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) * 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) * 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] * 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:40 sbisson@deploy1003: sbisson: Continuing with deployment * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 sbisson@deploy1003: sbisson: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:35 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] * 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] (duration: 13m 35s) * 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment * 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] * 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] (duration: 10m 29s) * 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.* * 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:11 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] * 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie * 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie * 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0 * 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts * 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s) * 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts * 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s) * 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0 * 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] (duration: 10m 09s) * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030 * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030 * 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030 * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] * 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030 * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029 * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029 * 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029 * 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet * 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:24 dzahn@dns1004: END - running authdns-update * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 17:22 dzahn@dns1004: START - running authdns-update * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed * 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] (duration: 13m 13s) * 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance * 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance * 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing * 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] * 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing * 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance * 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance * 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues * 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 15:50 moritzm: installing libhtml-parser-perl security updates * 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance * 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox * 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120 * 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120 * 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120 * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts * 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s) * 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s) * 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN [[phab:T436571|T436571]] * 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120 * 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002 * 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002 * 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 12:44 marostegui: Stop mariadb on db2250:s5 [[phab:T437411|T437411]] [[phab:T437279|T437279]] * 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] (duration: 33m 01s) * 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:08 kharlan@deploy1003: kharlan: Continuing with deployment * 12:06 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host (duration: 00m 54s) * 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host * 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host (duration: 01m 01s) * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host * 11:46 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] * 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] (duration: 13m 46s) * 11:34 kharlan@deploy1003: kharlan: Continuing with deployment * 11:33 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] * 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] (duration: 33m 09s) * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 11:13 kharlan@deploy1003: kharlan: Continuing with deployment * 11:12 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922 * 10:51 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] * 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie * 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 * 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 * 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 * 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 * 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 * 10:06 moritzm: installing libcap2 security updates * 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 * 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 * 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 * 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 * 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 * 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 * 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 * 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 * 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 * 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003 * 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 * 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues * 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues * 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie * 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - [[phab:T438828|T438828]] * 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues * 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues * 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations * 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet * 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:26 btullis@dns1004: END - running authdns-update * 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:23 btullis@dns1004: START - running authdns-update * 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet * 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 * 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 * 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3 * 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320 * 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320 * 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3 * 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1 * 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1 * 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:32 moritzm: installig zip security updates * 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:29 XioNoX: asw1-bw27-esams> request system reboot - [[phab:T437984|T437984]] * 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27 * 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27 * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:18 moritzm: installing gst-plugins-base1.0 security updates * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot * 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot * 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:23 awight: UTC morning deployment window complete * 07:22 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] (duration: 17m 46s) * 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment * 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] * 07:02 moritzm: installing pyasn1 security updates * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] (duration: 35m 52s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-21 == * 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:18 maryum: Deployed security fix for [[phab:T437708|T437708]] * 20:35 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] (duration: 15m 56s) * 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment * 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:19 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired * 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.* * 18:30 Emperor: repool eqiad sessionstore [[phab:T437915|T437915]] * 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired * 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie * 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm * 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie * 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.* * 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006 * 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie * 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie * 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] (duration: 12m 40s) * 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie * 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:36 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply * 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply * 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] * 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.* * 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie * 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.* * 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie * 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply * 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply * 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie * 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:13 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] (duration: 08m 09s) * 14:08 zabe@deploy1003: zabe: Continuing with deployment * 14:08 zabe@deploy1003: zabe: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie * 14:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 14:01 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] (duration: 12m 30s) * 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment * 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # [[phab:T438716|T438716]] * 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] (duration: 11m 40s) * 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment * 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] * 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] (duration: 14m 19s) * 13:15 mfossati@deploy1003: mfossati: Continuing with deployment * 13:14 mfossati@deploy1003: mfossati: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet * 13:07 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] * 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org * 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet * 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX) * 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet * 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet * 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet * 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # [[phab:T438499|T438499]] * 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet * 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie * 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:35 moritzm: installing chromium security updates * 09:22 tappof: bump space for prometheus k8s-dse in eqiad * 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s) * 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie * 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] * 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision * 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - [[phab:T438291|T438291]] * 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - [[phab:T438291|T438291]] * 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - [[phab:T438291|T438291]] * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw * 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet * 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003 * 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 [[phab:T435314|T435314]] * 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org == 2026-09-20 == * 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply == 2026-09-19 == * 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually ([[phab:T438591|T438591]], see [[phab:T438591|T438591]]#12341750 for what I did exactly) * 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> fr4h0g3thmx1uyhckys5893a4rwf86e 2466282 2466281 2026-10-11T02:01:26Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 04s) 2466282 wikitext text/x-wiki == 2026-10-11 == * 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 04s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-10 == * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 00:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 00:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync == 2026-10-09 == * 18:54 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] (duration: 09m 58s) * 18:49 cscott@deploy1003: cscott, ssastry: Continuing with deployment * 18:48 cscott@deploy1003: cscott, ssastry: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:44 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1354042{{!}}Undeploy Parsoid readviews on frwikisource (T440695)]] * 17:38 Daimona: Run database query from [[phab:T440557|T440557]]#12411029 in x1.wikishared * 16:47 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on netbox1003.eqiad.wmnet with reason: Restoring DB from backup on netboxdb1003 * 16:05 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:50 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:50 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge new dns entries for eqiad ab links - cmooney@cumin1004" * 15:46 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:44 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 15:44 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 15:42 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet * 15:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for wikikube-worker1057.eqiad.wmnet * 15:42 oblivian@cumin1004: START - Cookbook sre.hosts.remove-downtime for wikikube-worker1057.eqiad.wmnet * 14:40 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 14:40 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 14:39 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 14:33 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:30 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:25 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 14:24 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 14:20 ammarpad@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Abstract Wikipedia team' 'Abstract Wikipedia' Ammarpad --reason 'requested at [[:phab:T440606{{!}}T440606]]' * 14:16 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:59 elukey@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:57 fnegri@deploy1003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [eqiad] START helmfile.d/services/toolhub: apply * 13:56 fnegri@deploy1003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply * 13:55 fnegri@deploy1003: helmfile [codfw] START helmfile.d/services/toolhub: apply * 13:54 fnegri@deploy1003: helmfile [staging] DONE helmfile.d/services/toolhub: apply * 13:53 fnegri@deploy1003: helmfile [staging] START helmfile.d/services/toolhub: apply * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 13:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1120.eqiad.wmnet * 13:39 elukey@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sretest2003.codfw.wmnet'] * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1120.eqiad.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1120.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1120.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1119.eqiad.wmnet * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1119.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1119.eqiad.wmnet * 13:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1118.eqiad.wmnet * 13:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1118.eqiad.wmnet * 13:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1118.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1118.eqiad.wmnet * 13:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1118.eqiad.wmnet * 13:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1117.eqiad.wmnet * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need a few more things in place before we can reimage it in new location * 12:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1117.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:58 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1117.eqiad.wmnet * 12:58 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries for sretest1006 - cmooney@cumin1004" * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1117.eqiad.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1116.eqiad.wmnet * 12:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1116.eqiad.wmnet * 12:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 12:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1116.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1116.eqiad.wmnet * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1115.eqiad.wmnet * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1115.eqiad.wmnet * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1115.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1115.eqiad.wmnet * 12:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1115.eqiad.wmnet * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1114.eqiad.wmnet * 12:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1081.eqiad.wmnet with reason: hardware issues * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2374.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2373.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2372.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2371.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2370.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2369.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2368.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2367.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2366.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2365.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2364.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2363.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2362.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2361.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:30 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2360.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2359.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2358.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2357.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2356.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2355.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2354.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2353.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2352.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2351.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2350.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2349.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2348.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2347.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2346.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2345.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2344.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2343.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2342.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2341.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2340.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2339.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2338.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2337.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2336.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2335.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2334.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2333.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:29 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker2332.codfw.wmnet,dc=codfw,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1374.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1373.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:28 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1370.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1369.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1368.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1367.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1366.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1365.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1364.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1363.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1362.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1347.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1334.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1333.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1332.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1331.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1330.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1329.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:27 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:26 jayme@cumin1004: conftool action : set/weight=10:pooled=yes; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1114.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:25 jayme@cumin1004: conftool action : set/weight=10; selector: name=wikikube-worker1328.eqiad.wmnet,dc=eqiad,cluster=kubernetes,service=kubesvc * 12:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1114.eqiad.wmnet * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1113.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1113.eqiad.wmnet * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1113.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1113.eqiad.wmnet * 12:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1113-1120].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 11:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:56 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:53 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2902: test * 11:52 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2901: test * 11:42 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:38 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: test * 11:37 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1903: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1903: test * 11:35 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1902: test * 11:35 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1902: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 11:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 11:33 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 11:07 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:54 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 10:51 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:47 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:24 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . * 10:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1010.eqiad.wmnet with OS bookworm * 10:17 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on sretest1006.eqiad.wmnet with reason: move sretest1006 to row A * 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on lswtest-d8-eqiad,lswtest-d8-eqiad IPv6,ssw1-d[1,8]-eqiad with reason: take rail kit from test switch * 10:05 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 10:04 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:02 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cuminunpriv1001.eqiad.wmnet * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:02 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 10:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cuminunpriv1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:59 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:57 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 09:53 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on druid1010.eqiad.wmnet with reason: host reimage * 09:52 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 09:51 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 09:40 codders: Updated the Wikidata property suggester with data from 20261005's JSON dump * 09:37 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host druid1010.eqiad.wmnet with OS bookworm * 09:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 09:18 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cuminunpriv1001.eqiad.wmnet * 08:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 08:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1095.eqiad.wmnet * 08:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1095.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1095.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1094.eqiad.wmnet * 08:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1094.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1094.eqiad.wmnet * 08:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1093.eqiad.wmnet * 08:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1093.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1093.eqiad.wmnet * 08:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1087.eqiad.wmnet * 08:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1087.eqiad.wmnet * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1087.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1087.eqiad.wmnet * 07:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet * 07:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1086.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1086.eqiad.wmnet * 07:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1085.eqiad.wmnet * 07:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1085.eqiad.wmnet * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1085.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1085.eqiad.wmnet * 07:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1085.eqiad.wmnet * 07:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1084.eqiad.wmnet * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1084.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1084.eqiad.wmnet * 07:24 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 06:24 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:51 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) pool for host wikikube-worker1057.eqiad.wmnet * 05:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 05:42 oblivian@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 05:42 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:52 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 01:52 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 01:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 01:30 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] (duration: 10m 10s) * 01:25 kemayo@deploy1003: kemayo: Continuing with deployment * 01:24 kemayo@deploy1003: kemayo: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 01:20 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1353543{{!}}BaseEditCheck: don't use toDataElement when looking for categories (T440669)]] * 01:14 mutante: releases1003 - /usr/sbin/sysctl -w net.ipv4.tcp_congestion_control=bbr [[phab:T440068|T440068]] - downloads 120x times faster == 2026-10-08 == * 23:39 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning * 23:28 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) * 23:16 ejegg@deploy1003: ejegg: Continuing with deployment * 23:11 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:51 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353458{{!}}Update CentralNotice to cd3f46a4053cb74ff50b003e]] * 22:37 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) * 22:25 kemayo@deploy1003: kemayo: Continuing with deployment * 22:22 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 22:20 kemayo@deploy1003: kemayo: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 22:20 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 22:13 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 22:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 22:04 herron@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 22:03 herron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 22:02 herron@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 22:00 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 22:00 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1351444{{!}}VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] * 22:00 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 21:42 ejegg@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) * 21:37 ejegg@deploy1003: ejegg: Rolling back deployment * 21:27 ejegg@deploy1003: ejegg: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:15 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad * 21:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 21:14 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:13 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 21:07 ejegg@deploy1003: Started scap sync-world: Backport for [[gerrit:1353269{{!}}Update CentralNotice to 658422ff5fed886]] * 20:48 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) * 20:43 cjming@deploy1003: cjming: Continuing with deployment * 20:42 cjming@deploy1003: cjming: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:38 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1353392{{!}}Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] * 20:33 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] (duration: 09m 41s) * 20:27 mlitn@deploy1003: mlitn: Continuing with deployment * 20:27 mlitn@deploy1003: mlitn: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:23 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1353257{{!}}Temporary: don't catch exception]] * 20:21 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) * 20:14 cjming@deploy1003: milazg, cjming: Continuing with deployment * 20:10 cjming@deploy1003: milazg, cjming: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1339561{{!}}Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261{{!}}Update Test Kitchen instruments intake URL on beta cluster (T437720)]] * 19:58 kamila@deploy1003: Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) * 19:57 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:26 kamila@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 19:22 kamila@deploy1003: Started scap deploy-service shellbox: PHP update * 19:07 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 19:07 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 19:06 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 19:02 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 19:02 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 19:01 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet * 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet * 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet * 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet * 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet * 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet * 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes * 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet * 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts * 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s) * 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad * 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change * 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie * 16:07 swfrench@dns1004: END - running authdns-update * 16:04 swfrench@dns1004: START - running authdns-update * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) * 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage * 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie * 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet * 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet * 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet * 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning * 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet * 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet * 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet * 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes * 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet * 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet * 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:54 cmooney@dns3003: END - running authdns-update * 14:51 cmooney@dns3003: START - running authdns-update * 14:50 cgoubert@dns2004: END - running authdns-update * 14:48 cgoubert@dns2004: START - running authdns-update * 14:48 moritzm: installing pyasn1 security updates * 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet * 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet * 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" * 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] (duration: 36m 09s) * 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet * 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 14:23 moritzm: installing perl security updates * 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet * 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. * 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. * 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. * 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 * 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 14:03 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1353100{{!}}Bumping portals to master (T128546)]] * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet * 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet * 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:47 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) * 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet * 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet * 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment * 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet * 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet * 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet * 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet * 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet * 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet * 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:09 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1352515{{!}}Home Experience: Feature flag to enable Home (T439738)]] * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test * 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet * 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet * 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet * 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test * 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test * 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet * 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet * 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet * 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet * 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # [[phab:T440023|T440023]] [[phab:T439902|T439902]] * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet * 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet * 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet * 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet * 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet * 12:30 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) * 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet * 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet * 12:25 zabe@deploy1003: zabe: Continuing with deployment * 12:24 zabe@deploy1003: zabe: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet * 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet * 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1353155{{!}}Use Hadoop for UnusedTemplates on commonswiki (T422443)]] * 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet * 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet * 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet * 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet * 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet * 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet * 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet * 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] (duration: 04m 15s) * 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet * 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet * 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet * 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for [[phab:T417800|T417800]] * 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet * 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:10 topranks: cr1-codfw switch back to master routing engine [[phab:T426197|T426197]] * 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet * 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet * 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet * 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:36 topranks: cr1-codfw switch to backup routing engine [[phab:T426197|T426197]] * 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet * 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet * 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host * 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) * 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) * 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet * 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet * 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade * 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet * 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet * 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet * 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover * 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet * 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet * 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet * 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host * 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade [[phab:T417871|T417871]] * 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet * 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet * 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet * 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet * 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet * 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet * 08:59 moritzm: installing Wireshark security updates * 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet * 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet * 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover * 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover * 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json * 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json * 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - [[phab:T436922|T436922]] * 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 [[phab:T436922|T436922]]', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json * 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 [[phab:T436922|T436922]] * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet * 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet * 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet * 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet * 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet * 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:42 moritzm: installing python3.13 security updates * 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet * 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet * 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet * 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet * 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet * 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet * 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet * 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet * 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:01 moritzm: installing mbedtls security updates * 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet * 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet * 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet * 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet * 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet * 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet * 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet * 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) * 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet * 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet * 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet * 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment * 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet * 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet * 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet * 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352889{{!}}nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] * 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet * 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups * 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet * 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet * 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet * 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet * 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet * 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old ([[phab:T440358|T440358]]) * 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet * 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet * 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet * 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet * 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet * 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet * 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet * 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet * 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet * 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet * 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet * 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet * 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups * 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet * 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups * 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet * 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet * 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet * 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet * 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet * 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet * 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet * 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet * 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet * 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet * 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet * 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet * 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet * 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet * 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet * 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet * 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet * 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet * 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet * 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet * 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet * 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet * 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet * 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet * 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet * 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet * 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet * 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet * 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet * 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet * 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet * 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet * 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet * 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet * 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet * 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet * 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet * 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet * 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet * 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet * 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet * 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet * 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet * 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet * 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2141-2179].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet * 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font ([[phab:T393763|T393763]]) == 2026-10-07 == * 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet * 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm * 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage * 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply * 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply * 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply * 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" * 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet * 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) * 20:23 kemayo@deploy1003: kemayo: Continuing with deployment * 20:21 kemayo@deploy1003: kemayo: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:17 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1352600{{!}}EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] * 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply * 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply * 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply * 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet * 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet * 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet * 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet * 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet * 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet * 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet * 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet * 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet * 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts * 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s) * 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet * 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt * 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt * 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet * 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet * 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] (duration: 10m 40s) * 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki {{!}} {{Gerrit|1351442}} {{!}} [[phab:T432733|T432733]] * 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet * 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet * 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet * 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet * 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet * 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for [[phab:T417800|T417800]] (duration: 05m 23s) * 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:17 blake@deploy1003: Started scap sync-world: add restart policies for [[phab:T417800|T417800]] * 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet * 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet * 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) * 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet * 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) * 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet * 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet * 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) * 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) * 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade * 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade * 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet * 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet * 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet * 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting * 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade * 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade * 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade * 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade * 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade * 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade * 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade * 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade * 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) * 14:48 jforrester@deploy1003: jforrester: Continuing with deployment * 14:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams * 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams * 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs * 14:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352539{{!}}WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] * 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs * 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin * 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo * 14:34 jforrester@deploy1003: jforrester: Continuing with deployment * 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo * 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:27 moritzm: installing PHP 8.2 security updates * 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru * 14:26 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet * 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet * 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet * 14:22 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346643{{!}}Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531{{!}}Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] * 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 14:19 moritzm: installing postgresql-15 security updates * 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - [[phab:T439840|T439840]] * 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet * 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet * 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet * 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment * 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:54 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1352493{{!}}PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet * 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet * 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet * 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade * 13:44 papaul: start maintenance on cr2-codfw * 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) * 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet * 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet * 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2124-2140].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352456{{!}}Group fix for incorrect group mapping. (T439067)]] * 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) * 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment * 13:13 moritzm: installing mbedtls security updates * 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad * 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1352506{{!}}UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet * 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad * 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad * 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad * 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) * 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet * 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet * 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet * 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet * 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply * 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply * 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet * 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet * 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet * 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet * 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet * 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet * 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet * 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet * 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet * 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet * 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet * 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet * 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet * 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet * 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet * 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 andre: Rolling back again wmf.23 from group1 to group0 - [[phab:T438219|T438219]] * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet * 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet * 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet * 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P<nowiki>{</nowiki>lvs7003.magru.wmnet<nowiki>}</nowiki> and A:liberica * 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet * 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet * 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet * 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for [[phab:T417800|T417800]] (duration: 07m 32s) * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for [[phab:T417800|T417800]] * 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet * 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet * 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet * 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet * 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet * 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet * 09:54 duesen: just a test for SAL * 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:50 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) * 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet * 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet * 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:36 daniel@deploy1003: daniel: Continuing with deployment * 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet * 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet * 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet * 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:26 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet * 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet * 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet * 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet * 09:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 09:13 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352455{{!}}REST: Accept message objects as parameter help (T433534 T440377)]] * 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) * 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] * 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) * 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] * 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet * 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet * 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet * 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet * 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet * 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet * 08:20 aklapper@deploy1003: Rolling back deployment * 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet * 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet * 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet * 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet * 08:02 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) * 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment * 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet * 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet * 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet * 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet * 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia [[phab:T392465|T392465]] * 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet * 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:25 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1352357{{!}}Adds metrics for entity usage limit baseline (T439125)]] * 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. * 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. * 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 * 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. * 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet * 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. * 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet * 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet * 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. * 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. * 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet * 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet * 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet * 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet * 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. * 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. * 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet * 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:56 moritzm: installing bird security updates * 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet * 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet * 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet * 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet * 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet * 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet * 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 * 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet * 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet * 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet * 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet * 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet * 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet * 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet * 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet * 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet * 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet * 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning * 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet * 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet * 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet * 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet * 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet * 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet * 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet * 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet * 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:10 kart_: Updated Apertium to 2026-09-30-051826-production ([[phab:T437213|T437213]]) * 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet * 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply * 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply * 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply * 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply * 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host [[phab:T437279|T437279]] * 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning * 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host [[phab:T437279|T437279]] * 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host * 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply * 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-06 == * 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) * 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 22:00 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1352284{{!}}Revert "Scroll the TOC along with the page scrolling" (T440296)]] * 20:15 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 * 20:10 catrope@deploy1003: catrope: Continuing with deployment * 20:07 catrope@deploy1003: catrope: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt * 20:03 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1352217{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219{{!}}Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260{{!}}EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] * 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test * 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test * 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet * 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test * 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet * 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet * 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test * 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test * 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet * 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet * 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet * 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test * 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet * 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet * 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet * 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet * 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet * 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet * 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet * 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet * 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet * 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet * 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet * 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet * 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet * 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet * 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet * 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet * 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet * 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:39 bking@dns2004: END - running authdns-update * 16:37 bking@dns2004: START - running authdns-update * 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet * 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet * 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet * 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet * 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet * 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet * 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet * 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet * 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet * 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2055-2062,2064-2065].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 15:26 daniel@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) * 15:21 daniel@deploy1003: daniel: Continuing with deployment * 15:17 daniel@deploy1003: daniel: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing python3.13 security updates * 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 15:11 daniel@deploy1003: Started scap sync-world: Backport for [[gerrit:1352094{{!}}REST: enable the content/v2-beta module on testwiki (T433534)]] * 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw * 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] (duration: 00m 42s) * 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for [[phab:T440340|T440340]] * 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment * 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] (duration: 00m 51s) * 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for [[phab:T440340|T440340]] * 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment * 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 14:57 moritzm: restarting pybal on lvs1018 [[phab:T439730|T439730]] * 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet * 14:48 moritzm: restarting pybal on lvs1020 [[phab:T439730|T439730]] * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw * 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet * 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet * 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) * 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment * 14:33 moritzm: restarting pybal on lvs2012 [[phab:T439730|T439730]] * 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet * 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet * 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet * 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1325997{{!}}Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] * 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) * 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet * 14:23 moritzm: restarting pybal on lvs2014 [[phab:T439730|T439730]] * 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb * 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb * 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet * 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet * 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb * 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig * 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet * 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet * 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet * 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet * 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding chec}} * 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment * 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin * 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet * 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet * {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a s}} * 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet * 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352037{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036{{!}}refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035{{!}}fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031{{!}}refactor(AccountSetup): Move post-signup onboarding check}} * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment * 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1352025{{!}}ltwiki: Enable extended confirmed user group (T440216)]] * 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2033-2039].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 moritzm: installing php8.2 security updates * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 214m 10s) * 13:31 slyngshede@dns1004: END - running authdns-update * 13:29 slyngshede@dns1004: START - running authdns-update * 13:25 moritzm: installing libssh2 security updates * 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 13:14 moritzm: installing glibc security updates * 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' * 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 11:33 moritzm: installing chromium security updates * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply * 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply * 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage * 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 * 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back * 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" * 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor ([[phab:T406724|T406724]]) * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 * 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage * 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading * 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 [[phab:T436082|T436082]] * 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . * 10:09 moritzm: installing curl security updates * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing * 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache * 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing * 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m {{!}} tee gc_run_06102026` - [[phab:T437297|T437297]] * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . * 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 * 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] (duration: 00m 19s) * 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - [[phab:T437297|T437297]] * 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7 * 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back * 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply * 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply * 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D<nowiki>{</nowiki>wikikube-worker[2011-2018].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 * 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing * 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing * 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning * 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning * 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning * 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning * 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 * 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back * 08:00 matthiasmullie: UTC morning backport window done * 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) * 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:50 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1351378{{!}}Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379{{!}}Change lead section infobox exclusion (T439943)]], [[gerrit:1351518{{!}}[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] * 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) * 07:38 dcausse@deploy1003: dcausse: Continuing with deployment * 07:38 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test * 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test * 07:31 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351826{{!}}Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316{{!}}Find canonical pages on special page completion search (T440151)]] * 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 * 07:21 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) * 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back * 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment * 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343917{{!}}Config change for launch of stopping sending LL notifications into group1. (T437577)]] * 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 * 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7 * 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning * 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet * 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 * 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7 * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) * 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] (duration: 27m 17s) * 03:28 mwpresync@deploy1003: Rolling back deployment * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs [[phab:T438219|T438219]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-05 == * 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] (duration: 11m 59s) * 23:18 derenrich@deploy1003: derenrich: Continuing with deployment * 23:16 derenrich@deploy1003: derenrich: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 23:12 derenrich@deploy1003: Started scap sync-world: Backport for [[gerrit:1347724{{!}}Enable the discord previews from WikimediaCustomizations on enwiki (T439914)]] * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet * 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet * 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet * 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439719|T439719]] * 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439974|T439974]] * 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:51 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439932|T439932]] * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams * 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs * 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs * 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply * 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply * 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply * 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply * 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb) * 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo * 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply * 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply * 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # [[phab:T439577|T439577]] * 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s) * 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video * 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes * 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo * 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply * 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply * 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru * 16:32 dcausse: closing the "Distributed search queries experiment" window * 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] (duration: 10m 48s) * 16:25 dcausse@deploy1003: dcausse: Continuing with deployment * 16:23 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox * 16:19 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351216{{!}}Revert "search: test perf of moving completion out of eqiad" (T438984)]] * 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad * 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] (duration: 10m 44s) * 15:59 dcausse@deploy1003: dcausse: Continuing with deployment * 15:58 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:54 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351214{{!}}Revert "search: measure impact of enwiki" (T438984)]], [[gerrit:1351215{{!}}search: test perf of moving completion out of eqiad (T438984)]] * 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad * 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] (duration: 25m 12s) * 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm * 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet * 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox * 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet * 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet * 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage * 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:27 dcausse@deploy1003: dcausse: Continuing with deployment * 15:26 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet * 15:22 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351212{{!}}Revert "search: test impact of cross project searches" (T438984)]], [[gerrit:1351213{{!}}search: measure impact of enwiki (T438984)]] * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet * 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet * 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet * 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm * 15:16 elukey@dns1004: END - running authdns-update * 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] (duration: 10m 11s) * 15:14 elukey@dns1004: START - running authdns-update * 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet * 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet * 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:10 dcausse@deploy1003: dcausse: Continuing with deployment * 15:09 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:05 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351210{{!}}Revert "search: test impact of the function score window size" (T438984)]], [[gerrit:1351211{{!}}search: test impact of cross project searches (T438984)]] * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet * 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] (duration: 10m 12s) * 14:53 dcausse@deploy1003: dcausse: Continuing with deployment * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:52 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351223{{!}}Revert "search: test impact of MLR" (T438984)]], [[gerrit:1351209{{!}}search: test impact of the function score window size (T438984)]] * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply * 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet * 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet * 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet * 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet * 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply * 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts * 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet * 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s) * 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] (duration: 10m 04s) * 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet * 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet * 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet * 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet * 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:33 dcausse@deploy1003: dcausse: Continuing with deployment * 14:32 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:28 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351208{{!}}Revert "search: test impact of the MLR window size" (T438984)]], [[gerrit:1351222{{!}}search: test impact of MLR (T438984)]] * 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet * 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] (duration: 11m 12s) * 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet * 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:15 dcausse@deploy1003: dcausse: Continuing with deployment * 14:13 dcausse@deploy1003: dcausse: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 14:09 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1351207{{!}}search: test impact of the MLR window size (T438984)]] * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm * 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet * 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet * 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:58 logmsgbot: dreamyjazz Deployed security patch for [[phab:T439968|T439968]] * 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet * 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet * 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet * 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet * 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] (duration: 09m 59s) * 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service * 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet * 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:37 zabe@deploy1003: zabe: Continuing with deployment * 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service * 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:36 zabe@deploy1003: zabe: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet * 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet * 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1351194{{!}}Move reads/writes of wbc_entity_usage to x1 for (fr{{!}}ja)wiki (T439753)]] * 13:32 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] (duration: 11m 23s) * 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment * 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet * 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet * 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet * 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:21 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346053{{!}}Enable ReadingList CTA on Minerva for all reading list wikis (T438779)]], [[gerrit:1346793{{!}}Update pulsating blue dot behavior for anonymous bookmark (T438779)]] * 13:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] (duration: 14m 48s) * 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet * 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet * 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet * 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment * 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet * 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet * 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet * 13:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1349752{{!}}throttle.php: Lift IP cap for edit-a-thon (T439991)]] * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet * 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm * 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox * 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet * 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm * 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet * 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet * 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet * 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet * 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet * 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie * 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] (duration: 55m 26s) * 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:31 moritzm: installing sqlite3 security updates * 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment * 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:16 moritzm: installing mesa security updates * 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage * 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org * 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org * 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005 * 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005 * 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet * 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org * 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005 * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org * 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org * 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004" * 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005 * 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm * 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet * 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet * 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet * 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1350233{{!}}instrument(AccountSetup): expose motivation data for control (T439887)]] * 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet * 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet * 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet * 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet * 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet * 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet * 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm * 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet * 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet * 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet * 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet * 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004" * 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet * 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet * 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage * 10:46 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] (duration: 04m 44s) * 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:42 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Remove kafka-logging200[1-3] * 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet * 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet * 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing * 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests * 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing * 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests * 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm * 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests * 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests * 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing * 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing * 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet * 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet * 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing * 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm * 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests * 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests * 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet * 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet * 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie * 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for [[phab:T417800|T417800]] (duration: 09m 40s) * 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage * 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for [[phab:T417800|T417800]] * 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm * 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet * 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet * 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:22 volans: upgrading cloudcumin2001 in-place to trixie [[phab:T436667|T436667]] * 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm * 07:49 dpogorzelski@dns1004: END - running authdns-update * 07:46 dpogorzelski@dns1004: START - running authdns-update * 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-04 == * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-03 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-10-02 == * 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` [[phab:T440033|T440033]] * 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage * 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage * 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm * 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm * 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage * 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] * 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] * 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - [[phab:T427353|T427353]] * 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie * 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet * 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet * 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet * 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet * 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet * 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet * 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox * 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet * 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet * 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet * 17:44 dzahn@dns1004: END - running authdns-update * 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet * 17:41 dzahn@dns1004: START - running authdns-update * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet * 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet * 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 * 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 * 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors * 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" * 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm * 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) * 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox * 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 * 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" * 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox * 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 * 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet * 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet * 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes * 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) * 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm * 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes * 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet * 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet * 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm * 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm * 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet * 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. * 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm * 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot * 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) * 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) * 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) * 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox * 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm * 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm * 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" * 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox * 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage * 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts * 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm * 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" * 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox * 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART * 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) * 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts * 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-10-01 == * 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.* * 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp11[08,10,12,14].eqiad.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] (duration: 10m 19s) * 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1347806{{!}}WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)]] * 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — [[phab:T432444|T432444]] * 21:24 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] (duration: 11m 02s) * 21:19 egardner@deploy1003: egardner: Continuing with deployment * 21:17 egardner@deploy1003: egardner: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:13 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1347718{{!}}Preferred Sources: Move all eligibility checks to the client (T436692)]] * 21:06 brennen@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] (duration: 10m 43s) * 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment * 20:59 brennen@deploy1003: brennen, srapoj: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 brennen@deploy1003: Started scap sync-world: Backport for [[gerrit:1347692{{!}}Fix UTF-8 corruption in newline sanitization (T431960 T426631)]] * 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - [[phab:T432444|T432444]] * 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] (duration: 10m 56s) * 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:37 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1347686{{!}}Lose DB call for page prop (T439182)]], [[gerrit:1347687{{!}}Extract images in parser post-processing (T439182)]], [[gerrit:1347729{{!}}Make carousel open MMV directly (T439611)]], [[gerrit:1347735{{!}}Removes image preview from Treatment 1-2 (T439611)]] * 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:33 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] (duration: 18m 28s) * 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment * 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade. * 20:18 cjming@deploy1003: cjming, volker-e: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can * 20:14 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1347606{{!}}fix(ReadingRecommendations): Update button style to primary (T436332)]], [[gerrit:1347712{{!}}instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887)]], [[gerrit:1347743{{!}}instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)]] * 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two * 19:52 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] (duration: 09m 40s) * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:46 zabe@deploy1003: zabe: Continuing with deployment * 19:46 zabe@deploy1003: zabe: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:42 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347711{{!}}Add missing select of fr_timestamp]] * 19:42 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] (duration: 10m 01s) * 19:36 zabe@deploy1003: zabe: Continuing with deployment * 19:36 zabe@deploy1003: zabe: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 19:32 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347672{{!}}Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)]] * 19:19 jhathaway: puppet re-enabled * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689 * 18:25 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): logs kind of a mess, but no current blockers, rolling to all wikis * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s) * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply * 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply * 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox * 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T439100|T439100]] (duration: 33m 25s) * 17:16 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T439100|T439100]] * 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum * 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum * 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 () * 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:32 moritzm: installing socat security updates * 15:29 dpogorzelski@dns1004: END - running authdns-update * 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy * 15:25 dpogorzelski@dns1004: START - running authdns-update * 15:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] (duration: 10m 12s) * 15:14 zabe@deploy1003: zabe: Continuing with deployment * 15:14 zabe@deploy1003: zabe: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:11 moritzm: installing postgresql security updates * 15:09 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1347652{{!}}Add missing alias for filerevision]], [[gerrit:1347650{{!}}Add missing alias for filerevision]] * 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.* * 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.* * 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release * 14:48 moritzm: installing busybox bugfix updates from trixie point release * 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] (duration: 10m 05s) * 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment * 14:42 moritzm: installing zsh bugfix update from trixie point release * 14:41 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1347638{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]], [[gerrit:1347639{{!}}NotificationController: Use lazyPush() for the delete job (T439626)]] * 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release * 14:31 dpogorzelski@dns1004: END - running authdns-update * 14:29 dpogorzelski@dns1004: START - running authdns-update * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 14:04 awight: EU afternoon backport complete * 14:03 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] (duration: 10m 26s) * 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:58 awight@deploy1003: lilients, awight: Continuing with deployment * 13:57 awight@deploy1003: lilients, awight: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:53 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347483{{!}}Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)]] * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 13:41 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] (duration: 22m 53s) * 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:36 awight@deploy1003: awight, migr: Continuing with deployment * 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync * 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync * 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:23 awight@deploy1003: awight, migr: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve * 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 13:18 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1347422{{!}}instrument(AccountSetup): don't double count finish/skip clicks]], [[gerrit:1347423{{!}}stats(AccountSetup): Record onboarding ready time in Prometheus (T432810)]], [[gerrit:1347424{{!}}instrument(AccountSetup): Send exposure from server, send ready event (T432164)]] * 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply * 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s) * 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply * 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment * 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:34 moritzm: installing libtasn1-6 security updates * 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s) * 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:22 moritzm: installing glibc security updates * 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] * 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s) * 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] * 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply * 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply * 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad * 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] (duration: 12m 27s) * 11:50 jforrester@deploy1003: jforrester: Continuing with deployment * 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad * 11:47 jforrester@deploy1003: jforrester: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:43 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1347530{{!}}RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)]] * 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003" * 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. * 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. * 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw * 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw * 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] (duration: 10m 51s) * 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment * 10:44 javiermonton@deploy1003: javiermonton: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for [[gerrit:1346516{{!}}stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)]] * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # [[phab:T439805|T439805]] * 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply * 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply * 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply * 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s) * 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 * 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates * 08:05 moritzm: installing pcre2/openssl security updates * 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o) * 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] (duration: 14m 37s) * 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:16 kharlan@deploy1003: kharlan: Continuing with deployment * 07:15 kharlan@deploy1003: kharlan: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1346542{{!}}CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808)]], [[gerrit:1346541{{!}}CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)]] * 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] (duration: 33m 18s) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:54 kharlan@deploy1003: kharlan: Continuing with deployment * 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:53 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox * 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet * 06:33 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347223{{!}}Add extra UI elements to Special:AbuseReview following user feedback (T439694)]] * 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] (duration: 10m 33s) * 06:23 kharlan@deploy1003: kharlan: Continuing with deployment * 06:22 kharlan@deploy1003: kharlan: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 06:18 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1347210{{!}}AbuseReview: Send the abuse review tag in interaction events (T439238)]], [[gerrit:1347211{{!}}AbuseReview: Instrument when content policy match adds tags (T439242)]] == 2026-09-30 == * 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 21:22 sbassett: Deployed updated mitigation for [[phab:T435455|T435455]] * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) * 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:52 arlolra@deploy1003: arlolra: Continuing with deployment * 20:52 arlolra@deploy1003: arlolra: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:47 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1346763{{!}}Set a default for wgSplitParsoidParserCache in prod (T432760)]] * 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment * 20:31 mstyles@deploy1003: mstyles, catrope: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 20:11 mstyles@deploy1003: Started scap sync-world: Backport for [[gerrit:1346327{{!}}Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328{{!}}Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329{{!}}Email confirmation: Prepare for next experiment (T439283)]] * 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage * 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod * 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:22 brennen@deploy1003: Rolling back deployment * 18:10 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]): no current blockers, rolling to group1 * 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart * 17:15 bking@dns2004: END - running authdns-update * 17:13 bking@dns2004: START - running authdns-update * 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply * 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply * 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply * 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply * 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply * 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage * 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply * 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet * 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet * 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet * 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet * 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet * 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet * 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry * 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry * 14:54 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet * 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet * 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet * 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet * 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:49 zabe@deploy1003: zabe: Continuing with deployment * 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:48 zabe@deploy1003: zabe: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet * 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet * 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet * 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet * 14:44 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1346537{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538{{!}}Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] * 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet * 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet * 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet * 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet * 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet * 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet * 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) * 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:30 jforrester@deploy1003: jforrester: Continuing with deployment * 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet * 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker236[5-9].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:29 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet * 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet * 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:25 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346622{{!}}mlwiki: Enable Wikifunctions client mode]] * 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) * 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:18 jforrester@deploy1003: jforrester: Continuing with deployment * 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet * 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker133[0-4].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:16 jforrester@deploy1003: jforrester: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:12 jforrester@deploy1003: Started scap sync-world: Backport for [[gerrit:1346619{{!}}mlwiki: Enable Abstract Wikimedia client mode]] * 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] (duration: 11m 33s) * 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 Amir1: mwscript-k8s --comment="[[phab:T358266|T358266]]" --local_dblist=<(expanddblist all {{!}} grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing ([[phab:T358266|T358266]]) * 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary [[phab:T437937|T437937]] * 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment * 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1346611{{!}}Fix xsd xml version (T439093)]] * 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) * 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment * 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet * 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet * 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:697110{{!}}Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet * 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet * 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet * 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" * 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet * 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet * 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet * 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment * 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox * 13:25 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm * 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet * 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet * 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet * 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1345584{{!}}prv: Enable parsoid rendering for 5 wikis (T439444)]] * 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet * 13:18 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet * 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet * 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet * 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment * 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 13:10 samtar@deploy1003: migr, samtar: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet * 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker237[0-5].codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:06 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1346060{{!}}fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] * 13:05 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] (duration: 10m 41s) * 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART * 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:56 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - remove kafka-logging100[12] * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet * 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet * 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp7001.*<nowiki>}</nowiki> and A:cp - upgrade haproxy to 3.2.25-1 ([[phab:T439667|T439667]]) * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet * 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet * 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet * 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) * 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - [[phab:T439667|T439667]] * 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment * 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet * 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet * 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet * 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:26 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1328191{{!}}Profiler: Fix excimer component regex skipping the last frame in each stack]] * 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet * 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet * 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet * 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker138[0-3].eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] * 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet * 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet * 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet * 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet * 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet * 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply * 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply * 11:07 cgoubert@deploy1003: Finished scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] (duration: 11m 12s) * 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment * 10:59 cgoubert@deploy1003: cgoubert: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:56 cgoubert@deploy1003: Started scap sync-world: {{Gerrit|1330505}}: mediawiki: Redirect /api/ to /w/rest.php {{!}} [[phab:T433547|T433547]] * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet * 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet * 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service * 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet * 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D<nowiki>{</nowiki>wikikube-worker1384.eqiad.wmnet<nowiki>}</nowiki> and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) * 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - [[phab:T438454|T438454]] * 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:08 btullis@dns1004: END - running authdns-update * 09:06 btullis@dns1004: START - running authdns-update * 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' * 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' * 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad * 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad * 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca * 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw * 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw * 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:46 elukey@dns1004: END - running authdns-update * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . * 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 07:44 elukey@dns1004: START - running authdns-update * 07:35 moritzm: installing pcre2/openssl security updates * 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] (duration: 11m 27s) * 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 07:09 dcausse@deploy1003: dcausse: Continuing with deployment * 07:07 dcausse@deploy1003: dcausse: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:03 dcausse@deploy1003: Started scap sync-world: Backport for [[gerrit:1346012{{!}}Search: optimize morelike queries]] * 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm * 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet * 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet * 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-29 == * 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage * 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage * 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 * 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 * 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" * 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm * 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 * 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 * 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox * 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 * 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox * 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 * 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts * 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts * 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - [[phab:T427353|T427353]] [[phab:T395938|T395938]] * 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply * 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply * 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) * 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] * 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage * 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] * 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment * 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:55 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1344405{{!}}Beta Cluster: Use different Redis instance for LockManager (T436480)]] * 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm * 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) * 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 * 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 * 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:43 cklimas@deploy1003: cklimas: Continuing with deployment * 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" * 20:38 cklimas@deploy1003: cklimas: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:33 cklimas@deploy1003: Started scap sync-world: Backport for [[gerrit:1346087{{!}}InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] * 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox * 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 * 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) * 20:19 sbassett@deploy1003: sbassett: Rolling back deployment * 20:12 sbassett@deploy1003: sbassett: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm * 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 20:08 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1344774{{!}}Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649{{!}}Remove foundationwiki's CSP report-only header (T423691)]] * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm * 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" * 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage * 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage * 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage * 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply * 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply * 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm * 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm * 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors * 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.* * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" * 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm * 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" * 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet * 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet * 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 18:06 brennen: 1.47.0-wmf.22 train status ([[phab:T438218|T438218]]) - no current blockers, rolling to group0 * 18:05 sandraebele: Deployed Refinery as part of weekly deployment train * 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) * 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply * 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply * 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] * 17:53 sandraebele: Deploying Refinery as part of weekly deployment train * 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) * 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] * 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie * 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train * 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards * 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards * 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T420707|T420707]], [[phab:T420707|T420707]]: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards * 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.* * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host (duration: 00m 04s) * 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: [[phab:T420707|T420707]]: deploy to fresh wdqs host * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet * 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028 * 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` [[phab:T436186|T436186]] * 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # [[phab:T418194|T418194]] * 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply * 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply * 15:59 jhathaway: updating puppet stdlib, should be a noop * 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () * 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox * 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox * 14:21 moritzm: installing rsyslog security updates * 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet * 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet * 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw * 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) * 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" * 14:05 krinkle@deploy1003: krinkle: Continuing with deployment * 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox * 14:00 krinkle@deploy1003: krinkle: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary * 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary * 13:56 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1345216{{!}}File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298{{!}}ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] * 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] (duration: 00m 09s) * 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: [[phab:T331941|T331941]] * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet * 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie * 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) * 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment * 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345952{{!}}fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953{{!}}fix: prevent AccountSetup from being scrollable (T436872)]] * 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:25 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) * 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 13:20 samtar@deploy1003: samtar: Continuing with deployment * 13:19 samtar@deploy1003: samtar: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 13:14 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343958{{!}}CommonSettings: Load ModeratorToolkit extension (T438334)]] * 13:10 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) * 13:05 samtar@deploy1003: samtar: Continuing with deployment * 13:04 samtar@deploy1003: samtar: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:00 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343957{{!}}InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] * 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging * 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) * 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment * 12:44 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1345986{{!}}ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] * 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED * 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — [[phab:T432444|T432444]] * 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 * 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for [[phab:T417800|T417800]] (duration: 04m 14s) * 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for [[phab:T417800|T417800]] * 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage * 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # [[phab:T438232|T438232]] * 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie * 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" * 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox * 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet * 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # [[phab:T418194|T418194]] * 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) * 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment * 08:52 urbanecm@deploy1003: urbanecm: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344712{{!}}ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] * 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad * 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad * 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad * 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad * 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad * 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad * 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . * 08:10 moritzm: installing PHP 8.4 security updates * 07:49 matthiasmullie: UTC morning backport window done * 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) * 07:44 mlitn@deploy1003: mlitn: Continuing with deployment * 07:43 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:39 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345862{{!}}carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863{{!}}carousel: keep the hide button attached to the carousel (T439507)]] * 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) * 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet * 07:30 kharlan@deploy1003: kharlan: Continuing with deployment * 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet * 07:27 kharlan@deploy1003: kharlan: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:26 kart_: Updated cxserver to 2026-09-25-114231-production * 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply * 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply * 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply * 07:23 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1345854{{!}}CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] * 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply * 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org * 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) * 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org * 07:11 mlitn@deploy1003: mlitn: Continuing with deployment * 07:09 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now * 07:04 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345626{{!}}Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632{{!}}Cache extracted carousel data (T439182)]], [[gerrit:1345627{{!}}Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633{{!}}carousel: fix carousel image preload observer root (T439422)]] * 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply * 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply * 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply * 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply * 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm * 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm * 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage * 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage * 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035 * 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm * 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034 * 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm * 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm * 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure * 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage * 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 * 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033 * 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm * 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] (duration: 36m 03s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs [[phab:T438218|T438218]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie == 2026-09-28 == * 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:35 zabe: run script to fix fr_metadata drifts # [[phab:T420341|T420341]] * 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 21:47 sbassett: Deployed security mitigation for [[phab:T435455|T435455]] * 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () * 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync * 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync * 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync * 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - [[phab:T439201|T439201]] * 21:13 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) * 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment * 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 21:04 cjming@deploy1003: volker-e, cjming: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:00 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345624{{!}}fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643{{!}}fix(Homepage): apply the body transform only when an overlay is open (T439038)]] * 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) * 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia [[phab:T438375|T438375]] * 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 20:46 kemayo@deploy1003: kemayo: Continuing with deployment * 20:45 kemayo@deploy1003: kemayo: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:41 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1345608{{!}}BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637{{!}}EditCheck: only log a session if checks are actually possible (T438916)]] * 20:32 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) * 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie * 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment * 20:26 cjming@deploy1003: ssastry, cjming: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:22 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1345604{{!}}Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] * 20:16 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) * 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment * 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage * 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.* * 20:07 aude@deploy1003: aude, volker-e: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:03 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1345588{{!}}Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619{{!}}Enable Reading Recommendations experiment featured category setting (T437348)]] * 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie * 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts * 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s) * 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp601[3-6].drmrs.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () * 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... * 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts * 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s) * 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie * 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage * 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie * 16:20 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) * 16:15 zabe@deploy1003: zabe: Continuing with deployment * 16:10 zabe@deploy1003: zabe: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:06 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345298{{!}}wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] * 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () * 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] (duration: 19m 54s) * 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment * 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # [[phab:T439319|T439319]] * 15:45 jdrewniak@deploy1003: jdrewniak: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw * 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw * 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for [[gerrit:1345580{{!}}Bumping portals to master (T128546)]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw * 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 14:55 Lucas_WMDE: UTC afternoon backport+config window done * 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) * 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment * 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344771{{!}}feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772{{!}}feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" * 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) * 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox * 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment * 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345559{{!}}Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] * 13:58 vgutierrez: repool cp7001 - [[phab:T439361|T439361]] * 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) * 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment * 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - [[phab:T432444|T432444]] * 13:47 bking@dns2004: END - running authdns-update * 13:44 bking@dns2004: START - running authdns-update * 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1344216{{!}}Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] * 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw * 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) * 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 13:33 vgutierrez: depool cp7001 for some purged tests - [[phab:T439361|T439361]] * 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw * 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw * 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment * 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for [[gerrit:1345052{{!}}Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] * 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) * 13:11 sbisson@deploy1003: sbisson: Continuing with deployment * 13:07 sbisson@deploy1003: sbisson: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:03 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1345130{{!}}Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] * 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet * 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply * 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage * 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors * 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" * 11:28 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) * 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 * 11:22 zabe@deploy1003: zabe: Continuing with deployment * 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet * 11:21 zabe@deploy1003: zabe: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet * 11:17 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1345344{{!}}reconcileTables: Add sleep option (T438750)]] * 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet * 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet * 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply * 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply * 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 10:25 moritzm: installing chromium security updates * 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage * 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie * 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet * 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P<nowiki>{</nowiki>wikikube-worker2041.codfw.wmnet<nowiki>}</nowiki> and (A:wikikube-master-codfw or A:wikikube-worker-codfw) * 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage * 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" * 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 * 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet * 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet * 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet * 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor * 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync * 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync * 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage * 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 08:16 moritzm: installing Exim security updates * 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:59 matthiasmullie: UTC morning backport window complete * 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) * 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm * 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" * 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment * 07:49 mlitn@deploy1003: migr, mlitn: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 * 07:44 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1343611{{!}}Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594{{!}}EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] * 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] (duration: 32m 06s) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P<nowiki>{</nowiki>cp2059.*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 07:27 mlitn@deploy1003: mlitn: Continuing with deployment * 07:26 mlitn@deploy1003: mlitn: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi * 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1345070{{!}}[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371{{!}}Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369{{!}}Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370{{!}}Remove reliance on file repo (T439182)]] * 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-27 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-26 == * 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s) * 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 * 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] (duration: 10m 53s) * 08:02 oblivian@deploy1003: oblivian: Continuing with deployment * 08:00 oblivian@deploy1003: oblivian: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:56 oblivian@deploy1003: Started scap sync-world: Backport for [[gerrit:1345258{{!}}Revert "Disable Score exec"]] * 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply * 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply * 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply * 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply * 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply * 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] (duration: 11m 01s) * 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment * 03:30 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1345252{{!}}Disable Score exec (T439297 T438443)]] * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-25 == * 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm * 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values (duration: 00m 11s) * 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: [[phab:T439183|T439183]]: Accept dot, plus, hyphen in label values * 17:59 cdanis@dns1004: END - running authdns-update * 17:57 cdanis@dns1004: START - running authdns-update * 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.* * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie * 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet * 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet * 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet * 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage * 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - [[phab:T343000|T343000]] * 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.* * 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage * 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw * 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw * 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041 * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004" * 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # [[phab:T438750|T438750]] * 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet * 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041 * 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1019.*<nowiki>}</nowiki> and A:lvs * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet * 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm * 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P<nowiki>{</nowiki>lvs1020.*<nowiki>}</nowiki> and A:lvs * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040 * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004" * 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040 * 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active * 13:24 Emperor: repool sessionstore in codfw * 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet * 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage * 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004" * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox * 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039 * 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox * 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet * 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet * 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # [[phab:T438366|T438366]] * 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet * 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - [[phab:T439105|T439105]] * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet * 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet * 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox * 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet * 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.* * 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.* * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet == 2026-09-24 == * 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate * {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on t}} * 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment * 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment}} * {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for [[gerrit:1344049{{!}}ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050{{!}}ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118{{!}}Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560{{!}}Revert "Enable Reading Recommendations experiment on te}} * 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.* * 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet * 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet * 21:45 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s) * 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp4052.ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:40 catrope@deploy1003: catrope: Continuing with deployment * 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet * 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet * 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 21:33 catrope@deploy1003: catrope: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:28 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344795{{!}}Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] * 21:28 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] (duration: 19m 27s) * 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment * 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi * 21:08 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344406{{!}}ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716{{!}}ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740{{!}}Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752{{!}}Cap the dialog viewport (T439007)]] * 21:04 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s) * 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.* * 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.* * 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment * 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:53 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344750{{!}}Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329{{!}}prv: Enable parsoid rendering for 5 wikis (T438998)]] * 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:49 catrope@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s) * 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp405[1-2].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:44 catrope@deploy1003: catrope: Continuing with deployment * 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp404[1-6].ulsfo.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 20:43 catrope@deploy1003: catrope: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet * 20:39 catrope@deploy1003: Started scap sync-world: Backport for [[gerrit:1344388{{!}}HookHandler: Guard against recovery code expiry being null (T438593)]] * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s) * 20:14 kemayo@deploy1003: kemayo: Continuing with deployment * 20:12 kemayo@deploy1003: kemayo: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 20:08 kemayo@deploy1003: Started scap sync-world: Backport for [[gerrit:1344714{{!}}EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] * 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply * 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance * 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro * 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet` * 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet * 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness * 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance * 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly * 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance * 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw * 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance * 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters * 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian{{!}}tegola-vector-tiles) * 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s) * 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment * 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for [[gerrit:1344753{{!}}REST: restore PageContentHelper::checkAccess (fix live breakage)]] * 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance * 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance * 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance * 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance * 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw * 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance * 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance * 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance * 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance * 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability * 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 () * 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s) * 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified] * 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.* * 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw * 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus * 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.* * 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift * 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master{{!}}device-analytics{{!}}echostore{{!}}helm-charts{{!}}k8s-ingress-wikikube-ro{{!}}linkrecommendation{{!}}mathoid{{!}}restbase{{!}}restbase-async{{!}}rest-gateway-ro{{!}}mobileapps{{!}}mwdebug.*{{!}}push-notifications{{!}}recommendation-api{{!}}releases{{!}}wikifeeds * 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid{{!}}zotero * 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium{{!}}schema{{!}}termbox{{!}}proton{{!}}cxserver * 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor * 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.* * 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.* * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro * 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw * 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions * 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s) * 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo * 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw * 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo * 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet * 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie * 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.* * 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes * 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet * 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344711{{!}}AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693{{!}}Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] * 15:35 vgutierrez@dns1004: END - running authdns-update * 15:33 vgutierrez@dns1004: START - running authdns-update * 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s) * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply * 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment * 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet * 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344694{{!}}EventMapper::fetchByPage: Allow filtering by type (T438031)]] * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie * 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox * 15:06 vgutierrez@dns1004: END - running authdns-update * 15:04 vgutierrez@dns1004: START - running authdns-update * 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update * 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s) * 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.* * 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.* * 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:51 dkertesz: repooling cp7001{{!}}7011 after successful testing ([[phab:T343000|T343000]]) * 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 14:47 kharlan@deploy1003: kharlan: Continuing with deployment * 14:46 kharlan@deploy1003: kharlan: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet * 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet * 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.* * 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.* * 14:32 dkertesz: depooling cp7001{{!}}7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000) * 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1*.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1344684{{!}}AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677{{!}}AbuseReview: Inidicate if the queue hides recent edits (T438235)]] * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof * 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5025,5026].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:17 moritzm: installing Bird security updates * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie * 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.* * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s) * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie * 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344662{{!}}fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] * 13:40 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s) * 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:35 moritzm: installing nghttp2 security updates * 13:35 awight@deploy1003: awight: Continuing with deployment * 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage * 13:33 awight@deploy1003: awight: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1344246{{!}}Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'. * 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. * 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'. * 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s) * 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment * 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki * 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 mlitn@deploy1003: Started scap sync-world: Backport for [[gerrit:1344617{{!}}Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619{{!}}Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627{{!}}ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630{{!}}ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.* * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply * 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply * 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 11:13 hnowlan: restarted restbase on restbase2029 * 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[5031,5032].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:50 hnowlan: deleting stuck mw-web pods in eqiad * 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s) * 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision * 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344621{{!}}AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] * 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s) * 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision * 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344281{{!}}WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] * 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s) * 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw * 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 10:03 moritzm: installing apr-util security updates * 09:46 moritzm: installing bind9 security updates (client-side tools/libs only) * 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet * 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13 * 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13 * 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet * 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet * 09:01 XioNoX: asw1-b13-drmrs> request system reboot - [[phab:T437984|T437984]] * 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner * 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13 * 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:58 moritzm: installing node-lodash security updates * 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13 * 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision * 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12 * 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12 * 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade * 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt * 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade * 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet * 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet * 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1015.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 08:22 XioNoX: asw1-b12-drmrs> request system reboot - [[phab:T437984|T437984]] * 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12 * 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12 * 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner * 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade * 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade * 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, [[phab:T437984|T437984]]] * 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade * 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade * 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade * 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production ([[phab:T437213|T437213]]) * 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply * 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production * 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply * 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw * 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw * 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - [[phab:T439010|T439010]] * 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green * 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here * 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected ([[phab:T439010|T439010]]) * 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet * 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected ([[phab:T439010|T439010]]) * 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected ([[phab:T439010|T439010]]) * 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts * 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though * 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling) * 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state ([[phab:T439010|T439010]]) * 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.* * 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () == 2026-09-23 == * 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 23:56 brett: Switching acme-chief primary from codfw to eqiad - [[phab:T439010|T439010]] * 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7011.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones * 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P<nowiki>{</nowiki>cp7001.magru.wmnet<nowiki>}</nowiki> and A:cp - 7.1.1-2~bpo13+wmf3 () * 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia ([[phab:T438293|T438293]]) * 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident [[phab:T439010|T439010]]. Thursday UTC morning at the earliest, but please ask SRE oncall. * 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 121m 40s) * 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts ([[phab:T439010|T439010]]) * 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next * 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next * 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts ([[phab:T439010|T439010]]) * 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected ([[phab:T439010|T439010]]) * 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see * 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected ([[phab:T439010|T439010]]) * 22:24 ryankemper: [Cirrus] s/expected/expect * 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel * 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) * 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status * 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] (duration: 51m 29s) * 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 21:18 Emperor: ceph mgr fail on apus-be2005 * 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003 * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing * 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing * 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing * 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% * 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster * 20:49 swfrench@dns1004: END - running authdns-update * 20:46 swfrench@dns1004: START - running authdns-update * 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) * 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie * 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress [[phab:T439010|T439010]] * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 20:06 dzahn@dns1004: END - running authdns-update * 20:03 dzahn@dns1004: START - running authdns-update * 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie * 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie * 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 19:33 volans: rebooting arclamp2001.codfw.wmnet * 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on * 19:17 taavi@dns1004: END - running authdns-update * 19:14 taavi@dns1004: START - running authdns-update * 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.* * 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on * 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors * 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors * 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org * 18:45 sukhe@dns1004: END - running authdns-update * 18:43 sukhe@dns1004: START - running authdns-update * 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update * 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org * 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org * 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org * 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org * 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org * 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org * 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org * 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie * 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors * 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors * 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors * 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors * 18:01 cdanis@dns1005: END - running authdns-update * 17:58 cdanis@dns1005: START - running authdns-update * 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:54 taavi@dns1004: END - running authdns-update * 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage * 17:51 taavi@dns1004: START - running authdns-update * 17:46 taavi@dns1004: END - running authdns-update * 17:43 taavi@dns1004: START - running authdns-update * 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - [[phab:T439010|T439010]] * 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie * 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet * 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049 * 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage * 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 * 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage * 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie * 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie * 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie * 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:54 moritzm: installing cjose security updates * 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.* * 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s) * 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie * 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 15:46 dancy@deploy1003: Started scap sync-world: testing * 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie * 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie * 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook' * 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 [[phab:T438961|T438961]] * 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 * 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059 * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors * 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ [[phab:T438961|T438961]] * 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d [[phab:T438961|T438961]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before [[phab:T438786|T438786]] * 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie * 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox * 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 [[phab:T438961|T438961]] * 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply * 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie * 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply * 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply * 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply * 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply * 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad * 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 * 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts * 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad * 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s) * 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage * 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance * 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance * 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) * 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie * 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 * 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia [[phab:T438866|T438866]] * 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply * 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply * 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) * 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.* * 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply * 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply * 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1344292{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293{{!}}feat(AddLink): Do not resuggest an already reviewed page (T429417)]] * 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie * 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply * 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage * 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie * 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet * 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs-test1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 * 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 * 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 * 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:56 awight: manually run puppet agent * 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:45 awight: add seanleong-wmde to deployment-prep * 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad * 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 * 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) * 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 * 12:22 kharlan@deploy1003: kharlan: Continuing with deployment * 12:21 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] * 12:01 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343982{{!}}AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] * 11:51 Dreamy_Jazz: Deployed patch for [[phab:T438729|T438729]] * 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply * 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply * 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply * 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply * 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - [[phab:T438896|T438896]] * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply * 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web [[phab:T438896|T438896]] (duration: 04m 20s) * 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web [[phab:T438896|T438896]] * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet * 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet * 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet * 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet * 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-wdqs100[1-3].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) * 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd * 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 08:13 tappof@deploy1003: Finished scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] (duration: 12m 52s) * 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts * 08:04 tappof@deploy1003: Started scap sync-world: [[phab:T432444|T432444]] - Provision kafka-logging100[6-8] * 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 07:59 moritzm: installing giflib security updates * 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 07:29 moritzm: installing python-idna security updates * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance * 00:46 ryankemper: [WDQS] [[phab:T435443|T435443]] Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture * 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance == 2026-09-22 == * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet * 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet * 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet * 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet * 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet * 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet * 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet * 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet * 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet * 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] (duration: 13m 43s) * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet * 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet * 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet * 21:53 rzl@deploy1003: rzl: Continuing with deployment * 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet * 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 [[phab:T437403|T437403]] * 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s) * 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s) * 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet * 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet * 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet * 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet * 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s) * 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] * 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s) * 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] * 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s) * 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] (duration: 09m 53s) * 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] * 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:40 sbisson@deploy1003: sbisson: Continuing with deployment * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet * 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet * 20:40 sbisson@deploy1003: sbisson: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:35 sbisson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342285{{!}}Keep Article Guidance on where it is on today (T433293)]] * 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] (duration: 13m 35s) * 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet * 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment * 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1342825{{!}}eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)]] * 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] (duration: 10m 29s) * 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.* * 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment * 20:11 ebernhardson@deploy1003: ebernhardson: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for [[gerrit:1344014{{!}}cirrus: Send more_like traffic to eqiad]] * 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet * 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet * 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie * 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie * 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet * 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet * 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet * 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie * 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004" * 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet * 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie * 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet * 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet * 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet * 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage * 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0 * 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet * 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet * 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet * 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts * 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s) * 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts * 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet * 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet * 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet * 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s) * 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s) * 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet * 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet * 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet * 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet * 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet * 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet * 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet * 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet * 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0 * 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] (duration: 10m 09s) * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030 * 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030 * 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030 * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003" * 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment * 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344040{{!}}fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad]] * 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030 * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet * 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet * 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029 * 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029 * 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003" * 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox * 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029 * 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet * 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 17:24 dzahn@dns1004: END - running authdns-update * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet * 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet * 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet * 17:22 dzahn@dns1004: START - running authdns-update * 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet * 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet * 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet * 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet * 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw * 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet * 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet * 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet * 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed * 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet * 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet * 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] (duration: 13m 13s) * 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance * 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment * 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance * 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing * 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1344020{{!}}Enable AbuseReview on jawiki for likely PII (T438867)]] * 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing * 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet * 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet * 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet * 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet * 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance * 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance * 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues * 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet * 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet * 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet * 15:50 moritzm: installing libhtml-parser-perl security updates * 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet * 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004" * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance * 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance * 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance * 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox * 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet * 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet * 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet * 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120 * 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120 * 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120 * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet * 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet * 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet * 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet * 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet * 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet * 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts * 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s) * 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - [[phab:T435443|T435443]] * 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet * 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s) * 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified] * 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN [[phab:T436571|T436571]] * 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120 * 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120 * 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059 * 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059 * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet * 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet * 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet * 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet * 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet * 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet * 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet * 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker10[02-28].eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002 * 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002 * 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'. * 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'. * 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'. * 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet * 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet * 12:44 marostegui: Stop mariadb on db2250:s5 [[phab:T437411|T437411]] [[phab:T437279|T437279]] * 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet * 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet * 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet * 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] (duration: 33m 01s) * 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:08 kharlan@deploy1003: kharlan: Continuing with deployment * 12:06 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host (duration: 00m 54s) * 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to production host * 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host (duration: 01m 01s) * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet * 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>dse-k8s-worker1001.eqiad.wmnet<nowiki>}</nowiki> and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) * 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): [[phab:T435791|T435791]] to backup host * 11:46 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343953{{!}}AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)]] * 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] (duration: 13m 46s) * 11:34 kharlan@deploy1003: kharlan: Continuing with deployment * 11:33 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 11:27 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343960{{!}}AbuseReview: Hide Echo banner when user cannot see personal info (T438477)]] * 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] (duration: 33m 09s) * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 11:13 kharlan@deploy1003: kharlan: Continuing with deployment * 11:12 kharlan@deploy1003: kharlan: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922 * 10:51 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1343952{{!}}AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)]] * 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie * 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1 * 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1 * 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4 * 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4 * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8 * 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8 * 10:06 moritzm: installing libcap2 security updates * 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7 * 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7 * 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3 * 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3 * 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2 * 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2 * 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage * 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5 * 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5 * 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6 * 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003 * 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6 * 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia * 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues * 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues * 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P<nowiki>{</nowiki>cp[7010,7016].*<nowiki>}</nowiki> and A:cp - 3.2.23 upgrade ([[phab:T438828|T438828]]) * 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . * 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie * 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7 * 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - [[phab:T438828|T438828]] * 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues * 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues * 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations * 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet * 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 09:26 btullis@dns1004: END - running authdns-update * 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 09:23 btullis@dns1004: START - running authdns-update * 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet * 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues * 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7 * 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6 * 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6 * 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4 * 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3 * 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320 * 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320 * 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3 * 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1 * 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1 * 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply * 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply * 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply * 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply * 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt * 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts * 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 08:32 moritzm: installig zip security updates * 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:29 XioNoX: asw1-bw27-esams> request system reboot - [[phab:T437984|T437984]] * 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27 * 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad * 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27 * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 08:18 moritzm: installing gst-plugins-base1.0 security updates * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply * 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply * 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot * 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply * 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot * 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, [[phab:T437984|T437984]]] * 07:23 awight: UTC morning deployment window complete * 07:22 awight@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] (duration: 17m 46s) * 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment * 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:05 awight@deploy1003: Started scap sync-world: Backport for [[gerrit:1343347{{!}}Config change for launch of stopping sending LL notifications. (T438463)]], [[gerrit:1313951{{!}}Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)]] * 07:02 moritzm: installing pyasn1 security updates * 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s) * 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] (duration: 35m 52s) * 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs [[phab:T438217|T438217]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-09-21 == * 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:18 maryum: Deployed security fix for [[phab:T437708|T437708]] * 20:35 cjming@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] (duration: 15m 56s) * 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment * 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:19 cjming@deploy1003: Started scap sync-world: Backport for [[gerrit:1343100{{!}}Disable wgMFCustomSiteModules on German Wikipedia (T403380)]] * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply * 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired * 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.* * 18:30 Emperor: repool eqiad sessionstore [[phab:T437915|T437915]] * 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired * 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance * 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie * 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage * 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm * 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage * 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply * 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie * 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.* * 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006 * 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie * 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage * 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie * 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply * 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply * 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] (duration: 12m 40s) * 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment * 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie * 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage * 15:36 jdlrobson@deploy1003: jdlrobson: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply * 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply * 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for [[gerrit:1343579{{!}}Fixes: '.action_context' should be string (T437122)]] * 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.* * 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie * 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.* * 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . * 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply * 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply * 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie * 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply * 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply * 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie * 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage * 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 14:13 zabe@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] (duration: 08m 09s) * 14:08 zabe@deploy1003: zabe: Continuing with deployment * 14:08 zabe@deploy1003: zabe: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie * 14:05 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 14:01 zabe@deploy1003: Started scap sync-world: Backport for [[gerrit:1343542{{!}}Move wbc_entity_usage to x1 for mediawikiwiki (T438716)]], [[gerrit:1343556{{!}}Set db explicitly to false for virtual-wikibase-entityusage]] * 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] (duration: 12m 30s) * 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment * 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004" * 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # [[phab:T438716|T438716]] * 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1337580{{!}}nlwiki: enable SecurePoll local elections (T434045)]] * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply * 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox * 13:40 samtar@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] (duration: 11m 40s) * 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment * 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:29 samtar@deploy1003: Started scap sync-world: Backport for [[gerrit:1343319{{!}}arywiki: Create patroller and autopatrolled user groups (T438421)]] * 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] (duration: 14m 19s) * 13:15 mfossati@deploy1003: mfossati: Continuing with deployment * 13:14 mfossati@deploy1003: mfossati: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet * 13:07 mfossati@deploy1003: Started scap sync-world: Backport for [[gerrit:1343122{{!}}Let AA measure eligible readers w/o beta opt-in (T437076)]] * 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org * 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet * 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie * 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX) * 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply * 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply * 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply * 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply * 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply * 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply * 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply * 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply * 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply * 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply * 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply * 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply * 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply * 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply * 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART * 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply * 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet * 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet * 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet * 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet * 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # [[phab:T438499|T438499]] * 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet * 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet * 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request [[:phab:T438704{{!}}T438704]]' * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad * 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs * 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply * 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad * 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply * 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply * 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply * 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply * 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie * 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply * 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw * 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs * 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply * 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw * 09:35 moritzm: installing chromium security updates * 09:22 tappof: bump space for prometheus k8s-dse in eqiad * 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s) * 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage * 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie * 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [[gerrit:1341161{{!}}[Growth] Remove unused config variables (T392944)]] * 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision * 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - [[phab:T438291|T438291]] * 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - [[phab:T438291|T438291]] * 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - [[phab:T438291|T438291]] * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw * 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw * 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw * 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART * 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet * 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet * 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003 * 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 [[phab:T435314|T435314]] * 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org == 2026-09-20 == * 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply * 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply == 2026-09-19 == * 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually ([[phab:T438591|T438591]], see [[phab:T438591|T438591]]#12341750 for what I did exactly) * 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply * 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply * 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> jzt52xcoe4ez49k7p8a5l4ixrvonrna 498 443800 2466271 2464309 2026-10-10T14:31:07Z Stashbot 7414 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) 2466271 wikitext text/x-wiki === 2026-10-10 === * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> 44wbjpqmoiujdk2dnm1th3yynfspyi4 2466272 2466271 2026-10-10T14:31:18Z Stashbot 7414 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" 2466272 wikitext text/x-wiki === 2026-10-10 === * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> e7joms29lyay86oeq6pp7mxq09r7zfs 2466273 2466272 2026-10-10T14:31:36Z Stashbot 7414 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9926, 12139, 12882);" 2466273 wikitext text/x-wiki === 2026-10-10 === * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9926, 12139, 12882);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> pumyp33uzz3c6qub8ukkoeoke564uje 2466274 2466273 2026-10-10T14:31:48Z Stashbot 7414 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (7824, 13458, 13465, 13733, 13734, 13735, 13737);" 2466274 wikitext text/x-wiki === 2026-10-10 === * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (7824, 13458, 13465, 13733, 13734, 13735, 13737);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9926, 12139, 12882);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> 9l271ntsm5vc38gfizk410a6udd3ftr 2466275 2466274 2026-10-10T14:32:26Z Stashbot 7414 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9021, 9023, 9026, 9031, 9039, 9056, 9339, 9337, 9355, 9358, 9362, 9379, 9673, 9883, 9885, 9886, 9993, 11111, 13219, 13229, 13230, 13271, 2466275 wikitext text/x-wiki === 2026-10-10 === * 14:32 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9021, 9023, 9026, 9031, 9039, 9056, 9339, 9337, 9355, 9358, 9362, 9379, 9673, 9883, 9885, 9886, 9993, 11111, 13219, 13229, 13230, 13271, * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (7824, 13458, 13465, 13733, 13734, 13735, 13737);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9926, 12139, 12882);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> 6091smow5qb1cvxhujlg644phq031h6 2466276 2466275 2026-10-10T14:32:35Z Stashbot 7414 lucaswerkmeister: [cont.] 13283, 13301, 13434, 17412, 17445, 17477, 18154);" 2466276 wikitext text/x-wiki === 2026-10-10 === * 14:32 lucaswerkmeister: [cont.] 13283, 13301, 13434, 17412, 17445, 17477, 18154);" * 14:32 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9021, 9023, 9026, 9031, 9039, 9056, 9339, 9337, 9355, 9358, 9362, 9379, 9673, 9883, 9885, 9886, 9993, 11111, 13219, 13229, 13230, 13271, * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (7824, 13458, 13465, 13733, 13734, 13735, 13737);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9926, 12139, 12882);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> 6sn3hizsr5qwjniam8d0k2t4st6plth 2466277 2466276 2026-10-10T15:49:34Z Stashbot 7414 wmbot~lucaswerkmeister@tools-bastion-15: deployed 1461047..840114a: Move OAuth access token to separate database table (including db schema update) 2466277 wikitext text/x-wiki === 2026-10-10 === * 15:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed 1461047..840114a: Move OAuth access token to separate database table (including db schema update) * 14:32 lucaswerkmeister: [cont.] 13283, 13301, 13434, 17412, 17445, 17477, 18154);" * 14:32 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9021, 9023, 9026, 9031, 9039, 9056, 9339, 9337, 9355, 9358, 9362, 9379, 9673, 9883, 9885, 9886, 9993, 11111, 13219, 13229, 13230, 13271, * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (7824, 13458, 13465, 13733, 13734, 13735, 13737);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=1 WHERE background_batch IN (9926, 12139, 12882);" * 14:31 lucaswerkmeister: sql tools "UPDATE s53976__quickcategories.background SET background_auth=NULL, background_stopped_utc_timestamp=$(date +%s), background_stopped_localuser=5 WHERE background_batch IN (11648, 12156);" * 14:31 lucaswerkmeister: cleaning up some stale background runs in preparation for an upcoming migration to OAuth 2 (which will require OAuth tokens to be stored differently than they are now, with a uniqueness constraint that the stale rows violate) === 2026-10-06 === * 19:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1461047}}: Remove unused any_session() === 2026-09-20 === * 14:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|191f40dda7}} (remove service.template, unused since migraton to Toolforge Components Service) === 2026-08-25 === * 12:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5cc772f346}} (use MariaDB INSERT…RETURNING) === 2026-08-20 === * 23:14 wmbot~lucaswerkmeister@tools-bastion-15: rm -rf www-unused-since-2024-09-18/ # old source code, venv etc. – 130 MB / 124 MiB === 2026-08-18 === * 12:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42e38be327}} via push-to-deploy === 2026-08-17 === * 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe67524e52}} (migrate to Toolforge Components Service for webservice and backround runner) === 2026-05-25 === * 13:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e867fb34a0}} (support proofread-page content model) * 11:42 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fdeac4ae23}} (allow sourceswiki / multilingual Wikisource, also remove Wikinews) === 2026-04-06 === * 23:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d83be52ab9}} (reenable notifications feature with new OAuth consumer: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-04-03 === * 11:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1079b511fd}} (refactor authentication_area()) * 11:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb7875d76b}} (temporarily disable notifications feature to unbreak tool: [[phab:T420154|T420154]], [[phab:T421991|T421991]]) === 2026-03-15 === * 19:48 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0e50ebab0d}} (update docs links) === 2026-01-12 === * 17:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b6cd1e9722}} (handle mwoauth-invalid-authorization) * 13:11 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c01aa71aef}} (fix application context lifecycle errors) === 2026-01-11 === * 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13cb4c4bad}} (support creating missing pages) === 2025-12-20 === * 19:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7719b7b594}} (refactor: kw-only param) * 18:49 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68012a14ac}} (upgrade dependencies) === 2025-09-03 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: sql tools <<< 'DROP TABLE s53976__quickcategories.scratch' # unused and empty, presumably I was experimenting with auto_increment behavior here years ago === 2025-08-24 === * 22:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|597636792a}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility) === 2025-07-15 === * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1672ea1585}} (fix dev-requirements.in) === 2025-07-13 === * 11:18 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80ed3ddf01}} (split dev requirements from prod requirements → smaller image, faster build) * 11:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|060c14930c}} (upgrade dependencies) === 2025-07-07 === * 07:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b65c99d03}} (upgrade dependencies, including mwparserfromhell 0.7.2) === 2025-06-18 === * 18:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f93390eb6}} (&actions= and &title= parameters in /batch/new/pagepile, [[phab:T397320|T397320]]) === 2025-06-11 === * 17:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|56f47f15b7}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]]) === 2025-05-22 === * 21:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|74cd3dee83}} (install PyMySQL from Git for Python 3.13 compatibility; CC [[phab:T381923|T381923]]) * 21:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1d0fb31941}} (upgrade dependencies) === 2025-05-12 === * 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d949e8ee4a}} (Python 3.13 using --use-latest-versions via [[phab:T381923|T381923]]) === 2025-02-24 === * 19:08 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars delete TOOL_EXPECTED_DATABASE_ERROR && webservice restart === 2025-02-20 === * 13:36 lucaswerkmeister: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href="https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/AYLIOWBNA7KB6YJERKEJ7TMN6WR2H2I5/">database maintenance</a>.' && webservice restart === 2025-02-03 === * 17:28 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EXPECTED_DATABASE_ERROR 'The tool is temporarily non-functional due to <a href=https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/QB2WKVCBOPUUXDXMEKREGX4HQSU7JZ4P/>necessary database maintenance</a>.' && webservice restart === 2024-12-16 === * 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|fa4f0b15c3}} (cache db connection in application context [i.e. only for the duration of one web request]) === 2024-12-03 === * 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7dc5c6777e}} (add database index) === 2024-12-01 === * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6cd8c4e34}} (add preference for watchlist behavior) === 2024-11-29 === * 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|995de58bee}} (remove unused retry_id column) === 2024-11-24 === * 15:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|91ee3ebea2}} (upgrade dependencies, including Flask 3.1) === 2024-10-25 === * 19:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13fc15a961}} (upgrade dependencies, including Werkzeug 3.0.6) === 2024-10-13 === * 11:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2a72076fda}} (upgrade dependencies, including MarkupSafe 3.0); also, disregard the previous SAL message (2024-10-11) which was in fact meant for the wd-image-positions tool (there’s no l10n in quickcategories yet) – I accidentally deployed and logged in the wrong tool (the deploy presumably introduced no changes) ^^ === 2024-10-11 === * 17:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) === 2024-10-10 === * 21:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|119ab5275b}} (handle bad content format / model errors better) === 2024-09-28 === * 13:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6ad9841d7}} (update dependencies) * 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3edc9faa4e}} (improve typing: tokyo drift) * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2be9737e71}} (2 improve 2 typing) * 12:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|03c567ef7f}} (further improve typing) * 11:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f5f9683eb}} (improve typing, should be a no-op) === 2024-09-18 === * 20:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b9a658f45e}} (health check for background runner, [[phab:T374152|T374152]]) * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: mv www/ www-unused-since-2024-09-18/ * 19:47 wmbot~lucaswerkmeister@tools-bastion-13: curl -sL 'https://gitlab.wikimedia.org/toolforge-repos/quickcategories/-/raw/main/service.template' > service.template * 19:03 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && toolforge jobs run background-runner --command background-runner --image tool-quickcategories/tool-quickcategories:latest --continuous * 19:00 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop && webservice --mount=none buildservice start * 18:44 lucaswerkmeister: toolforge build start https://gitlab.wikimedia.org/toolforge-repos/quickcategories === 2024-09-17 === * 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4cfe24a186}} (fix background runner database config) * 19:58 wmbot~lucaswerkmeister@tools-bastion-13: kubectl delete deployment background-runner && kubectl create -f deployment.yaml # recreate background runner, seemed to be running old code version? idk * 19:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0be765159c}} (editgroups from envvars) * 19:53 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__SINCE 2021-09-14T00:00:00Z * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__URL "'https://editgroups-commons.toolforge.org/b/QC/<nowiki>{</nowiki>0<nowiki>}</nowiki>/'" * 19:52 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_EDITGROUPS__COMMONSWIKI__DOMAIN commons.wikimedia.org === 2024-09-16 === * 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a82f584c44}} (read toolsdb from envvars); commented out DATABASE section in config.yaml, should use envvars instead * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__DB s53976__quickcategories * 19:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge envvars create TOOL_DATABASE__TOOLSDB true === 2024-09-15 === * 19:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|377aab02c6}} (lowercase nested keys to work around [[phab:T374780|T374780]]) * 19:25 lucaswerkmeister: commented out OAUTH section in config.yaml, should use envvars instead * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_secret"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET * 19:25 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["OAUTH"]["consumer_key"]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY * 14:12 wmbot~lucaswerkmeister@tools-bastion-13: commented out SUMMARY_BATCH_LINK in config.yaml, TOOL_SUMMARY_BATCH_LINK envvar is now used instead * 14:11 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open("config.yaml"))["SUMMARY_BATCH_LINK"]))' {{!}} toolforge envvars create TOOL_SUMMARY_BATCH_LINK * 12:44 wmbot~lucaswerkmeister@tools-bastion-13: removed SUMMARY_SUFFIX from config.yaml, unused since {{Gerrit|da859e4b81}} five years ago === 2024-09-14 === * 18:33 wmbot~lucaswerkmeister@tools-bastion-13: commented out SECRET_KEY in config.yaml, TOOL_SECRET_KEY envvar is now used instead * 18:31 lucaswerkmeister: toolforge envvars create TOOL_SECRET_KEY "$(python3 -c 'import yaml; print(yaml.safe_load(open("config.yaml"))["SECRET_KEY"])')" * 18:26 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c2dfddb8b1}} (load config from TOOL_* envvars in addition to config.yaml; no envvars actually added yet) * 17:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|feae0e7e66}} (refactor config) * 13:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6d6aed57e0}} (update dependencies); includes a restart of the background runner which probably “fixes” [[phab:T374152|T374152]] for now === 2024-08-09 === * 09:15 wmbot~lucaswerkmeister@tools-bastion-13: kubectl rollout restart deployment background-runner # seemed stuck? === 2024-06-12 === * 20:28 wmbot~lucaswerkmeister@tools-bastion-13: restarted webservice to pull in new image for [[phab:T367345|T367345]] === 2024-04-08 === * 19:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|5dbb79bd6f}} (configure health-check-path) * 18:18 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|2ed8a895fd}} (make session permanent after login) === 2024-01-04 === * 12:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3d120692b5}} (update dependencies, mwparserfromhell 0.6.6) === 2023-12-28 === * 13:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ade8d9c639}} (better whitespace cleanup) * 11:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0ad1812b77}} (allow Wikifunctions) === 2023-10-25 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d817ca5f60}} (Werkzeug 3.0.1) === 2023-10-03 === * 12:19 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d174a2b28c}} (support ProofreadPage index pages; should resolve background-runner crash loop) * 11:12 wm-bot: <lucaswerkmeister> deployed {{Gerrit|23486f8772}} (update dependencies, Flask+Werkzeug 3) === 2023-09-07 === * 19:52 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be8e6bd4}} (upgrade dependencies; Flask/Werkzeug, mwparserfromhell – nothing major) === 2023-07-15 === * 13:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a937a30991}} (cleanup typings and update github actions) * 13:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7fd188d540}} (Python 3.11) === 2023-05-01 === * 23:47 wm-bot: <lucaswerkmeister> deployed {{Gerrit|18060ebfea}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq) === 2023-04-29 === * 15:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b06fded200}} (update dependencies, Flask/Werkzeug 2.3) === 2023-04-06 === * 19:12 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again; also restarted both webservice and background-runner in case it was needed to pick up the new DB server (I didn’t check) * 16:17 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB maintenance === 2023-04-05 === * 19:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e27a219ad0}} (remove some wmflabs references) === 2023-04-01 === * 14:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bfbe83dfb4}} (slightly improve some mwapi calls) === 2023-03-28 === * 19:11 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR again === 2023-03-27 === * 16:39 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR ahead of tomorrow’s WMCS maintenance === 2023-03-22 === * 23:41 wm-bot: <lucaswerkmeister> unset EXPECTED_DATABASE_ERROR – according to T.329970#{{Gerrit|8679480}}, setting up a replica of CloudDB was finally successful (🥳), so no more database errors are expected in the near future === 2023-02-21 === * 00:43 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 00:43 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2533068}} rows (~tools.quickcategories/purge-querytime) === 2023-02-20 === * 23:29 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 23:28 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 23:26 wm-bot: <lucaswerkmeister> deployed {{Gerrit|613fd8d548}} (disable querytime mechanism by default: wrote much data for little gain) * 10:35 wm-bot: <lucaswerkmeister> set EXPECTED_DATABASE_ERROR for upcoming ToolsDB downtime === 2023-02-14 === * 21:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3748a8600e}} (fix empty titles in runner; hopefully resolves the CrashLoopBackOff which was at 21487 restarts 😬) * 20:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|011455de8f}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes) === 2022-09-10 === * 18:42 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a573b88e34}} (README fix, pulled without webservice restart) * 18:33 wm-bot: <lucaswerkmeister> deployed {{Gerrit|306045651d}} (diffusion → gitlab) === 2022-06-24 === * 15:53 wm-bot: <lucaswerkmeister> deployed {{Gerrit|64df38cf96}} (README.md update – pulled without webservice restart) === 2022-04-19 === * 16:29 wm-bot: <lucaswerkmeister> removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over === 2022-04-13 === * 20:18 wm-bot: <lucaswerkmeister> optimized querytime table (~tools.quickcategories/purge-querytime) * 20:18 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches: deleted {{Gerrit|2085455}} rows (~tools.quickcategories/purge-querytime) * 19:04 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (~tools.quickcategories/purge-querytime) * 18:30 wm-bot: <lucaswerkmeister> added EXPECTED_DATABASE_ERROR to config so users will see an informative error message during next week’s ToolsDB outage === 2022-03-26 === * 17:50 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c934f7d8ae}} (remove canonical: True from service.template) without restart since it should be a no-op * 17:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|45239001ac}} (pip-tools) === 2022-02-15 === * 19:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3627eb9988}} (fix query) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bb178f29b2}} (missing template files) * 19:05 wm-bot: <lucaswerkmeister> restarted webservice with `kubectl rollout restart deployment quickcategories`, the `webservice restart` apparently didn’t work? (pod age was still 125d) * 19:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6fc4cf0508}} (recognize interwiki titles) with background runner restart, seems to be going better now * 18:40 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3db46cd631}} (update for cachetools 5.0.0), reset all pending jobs to planned, recreated background runner * 18:21 wm-bot: <lucaswerkmeister> deleted background-runner deployment again, I have a stack trace, let’s see if I can make sense of it * 18:18 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.4) and recreated background-runner deployment from scratch (it had been stuck in CrashLoopBackOff at 12845 restarts, let’s see if this helps) === 2021-10-13 === * 18:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7a5d6823e2}} (remove type ignore comments), updated dependencies including Flask 2.0.2 === 2021-09-25 === * 14:48 wm-bot: <lucaswerkmeister> removed old venv-3.7 === 2021-09-18 === * 13:02 wm-bot: <lucaswerkmeister> deployed {{Gerrit|0213175db9}} (migrate to dataclasses and abc) === 2021-09-14 === * 22:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e8a95d4c04}} (background runner speedup) * 22:32 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8ddb2092b2}} (Python 3.9, i.e. new venv) * 20:43 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6d375c7dcc}} (link to Commons edit groups) === 2021-09-03 === * 16:05 wm-bot: <lucaswerkmeister> updated venv (includes mwparserfromhell 0.6.3) === 2021-08-08 === * 22:08 wm-bot: <lucaswerkmeister> END - optimize querytime table * 22:08 wm-bot: <lucaswerkmeister> START - optimize querytime table * 22:02 wm-bot: <lucaswerkmeister> END - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches (deleted {{Gerrit|10257486}} rows) * 13:16 wm-bot: <lucaswerkmeister> START - purge querytime rows older than 30 days, in batches of 1000 sleeping for 1s between batches === 2021-08-07 === * 20:13 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7028c292e7}} (lock less tables) * 19:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|bbdebedd08}} (styling improvement) === 2021-07-20 === * 19:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1d17bc8e87}} (update config loading code) * 19:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1f23a7801}} (minor updates) === 2021-05-30 === * 15:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|9618989c17}} (rename deployment from quickcategories.background-runner to just background-runner) * 15:14 wm-bot: <lucaswerkmeister> restarted background runner as well (turns out the deployment.yaml in ~ was old and I should’ve used the www/python/src/ one) * 15:12 wm-bot: <lucaswerkmeister> started webservice again (background runner not yet, odd kubectl error) * 15:12 wm-bot: <lucaswerkmeister> updated dependencies (e.g. mwparserfromhell 0.6.2, flask 2.0.1) * 15:10 wm-bot: <lucaswerkmeister> (this includes the background runner as well) * 15:09 wm-bot: <lucaswerkmeister> temporarily stopping webservice for dependency update (seems like NFS is preventing a pip upgrade while the tool is running?) === 2021-04-09 === * 18:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|24f6e19113}} (better workaround for [[phab:T279585|T279585]]) === 2021-04-07 === * 20:01 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c3c66caf96}} (work around [[phab:T279585|T279585]]) === 2021-02-28 === * 19:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|03d707756b}} (fix return type, should be a no-op) * 19:15 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a01dae7728}} (better OAuth error handling) === 2021-02-26 === * 10:28 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e7654cf4b3}} (link PagePile batch creation from index) === 2021-02-24 === * 21:51 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b2f83f22c0}} (a11y fix) === 2021-02-16 === * 20:29 wm-bot: <lucaswerkmeister> deployed {{Gerrit|cc50c3ee7a}} (add skip link) * 19:46 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1265f0d223}} (Bootstrap update) === 2020-12-28 === * 09:00 wm-bot: <lucaswerkmeister> updated Python packages === 2020-12-17 === * 17:24 wm-bot: <lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over === 2020-12-16 === * 16:44 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e236b28c74}} (error handler for upcoming ToolsDB maintenance) === 2020-12-14 === * 20:21 wm-bot: <lucaswerkmeister> deployed {{Gerrit|8c9a2dfc59}} (fix current_url) === 2020-11-14 === * 22:07 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f10c62ced1}} (new export mode) === 2020-11-10 === * 15:57 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-11-02 === * 16:17 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled (again) * 15:56 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-27 === * 16:25 wm-bot: <lucaswerkmeister> tool is read-write again, ToolsDB maintenance was rescheduled * 15:55 wm-bot: <lucaswerkmeister> made tool readonly to avoid issues during ToolsDB maintenance === 2020-10-17 === * 15:14 wm-bot: <lucaswerkmeister> deployed {{Gerrit|d5609a1b7f}} (more durable CSRF tokens; no background runner restart) === 2020-06-26 === * 21:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e1c86c5b27}} (update pagepile url) === 2020-06-15 === * 20:47 wm-bot: <lucaswerkmeister> renamed default branch from master to main === 2020-06-14 === * 11:16 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c0be64fafd}} (layout fix) === 2020-04-25 === * 17:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|90342028c5}} (toolforge.org) === 2020-02-20 === * 23:36 wm-bot: <lucaswerkmeister> deployed {{Gerrit|1f063050d9}} (many code cleanups including some minor bugfixes) === 2020-02-16 === * 14:56 lucaswerkmeister: deployed {{Gerrit|5e1e3d97ac}} (follow redirects by default, !title to edit the redirect itself) === 2020-02-02 === * 17:18 wm-bot: <lucaswerkmeister> deployed {{Gerrit|db2a7c16ae}} (Python 3.7, 2020 Kubernetes cluster), includes venv rebuild; tool had apparently been broken before for about half a month === 2019-11-08 === * 17:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32bb3fcae6}} (silence bs4 warning) === 2019-09-08 === * 20:22 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b99c878d2a}} (db schema cleanup) * 19:35 wm-bot: <lucaswerkmeister> reset command 266888 (stuck in pending status) and restarted background runner === 2019-09-02 === * 20:05 wm-bot: <lucaswerkmeister> deployed {{Gerrit|c601f81355}} (retry earlier after wiki read-only) === 2019-08-31 === * 18:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|727e5dc3f7}} (clarify retry and CSRF) === 2019-08-08 === * 17:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|6b45be2605}} (fix wiki read-only display) === 2019-06-30 === * 22:24 wm-bot: <lucaswerkmeister> deployed {{Gerrit|b1a75f69dc}} (remove debug code) * 22:09 wm-bot: <lucaswerkmeister> deployed {{Gerrit|20a47a955b}} (preferences linked in navbar) * 18:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|833372331d}} (preferences – not linked anywhere yet – with experimental OOUI version) === 2019-06-19 === * 11:55 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a35b558374}} (add logout route) === 2019-06-18 === * 20:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|99f9d3f4c2}} (copy nav below lists) === 2019-06-16 === * 14:37 wm-bot: <lucaswerkmeister> deployed temporary change (hide notifications count for Harmonia Amanda per request) * 14:11 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a415a923dd}} (show notifications count) === 2019-06-01 === * 20:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|32b8528d58}} (export options, including PagePile export) * 12:04 wm-bot: <lucaswerkmeister> deployed {{Gerrit|7ba32d514c}} (PagePile support) === 2019-05-31 === * 22:06 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ebfed74416}} (minor error handling improvements) * 16:57 wm-bot: <lucaswerkmeister> deployed {{Gerrit|3c58cc0810}} (minor UI improvements) === 2019-05-30 === * 22:30 wm-bot: <lucaswerkmeister> deployed {{Gerrit|ff3e7b2931}} (add sort key hint to placeholder) * 22:17 wm-bot: <lucaswerkmeister> deployed {{Gerrit|e3cd2871eb}} (sort key support and minor UI fix) * 22:12 wm-bot: <lucaswerkmeister> git remote add github https://github.com/lucaswerkmeister/tool-quickcategories.git # work around [[phab:T224677|T224677]] === 2019-05-18 === * 21:39 wm-bot: <lucaswerkmeister> deployed {{Gerrit|13946ed630}} (drop Python 3.4 compatibility) * 20:52 wm-bot: <lucaswerkmeister> rebuilt webservice venv for Python 3.5 * 20:46 wm-bot: <lucaswerkmeister> switch webservice to python3.5 * 20:34 wm-bot: <lucaswerkmeister> deployed {{Gerrit|a0a6530cfe}} (rearrange buttons) * 09:37 wm-bot: <lucaswerkmeister> deployed {{Gerrit|83d01bb93b}} (background UI improvements) === 2019-05-17 === * 21:45 wm-bot: <lucaswerkmeister> deployed {{Gerrit|57faa03444}} (fix tabindex) === 2019-05-16 === * 18:58 wm-bot: <lucaswerkmeister> deployed {{Gerrit|f53ae8ac71}} (remove commands help from index page) === 2019-05-15 === * 20:24 wm-bot: Toolforge dologmsg test ([[phab:T222244|T222244]]) === 2019-05-12 === * 13:44 lucaswerkmeister: deployed {{Gerrit|1bbd058bb6}} (minor fix) * 13:15 lucaswerkmeister: deployed {{Gerrit|cfb2b6975a}} (query times HTML improvements) === 2019-05-11 === * 22:56 lucaswerkmeister: deployed {{Gerrit|538f52a1db}} (handle invalid titles) * 20:55 lucaswerkmeister: deployed {{Gerrit|89161adc8a}} (show query execution times) === 2019-05-09 === * 23:24 lucaswerkmeister: deployed {{Gerrit|cf208a54e9}} (track query execution times) === 2019-05-04 === * 20:53 lucaswerkmeister: deployed {{Gerrit|8a3ff068e2}} (HTML improvements) * 20:52 lucaswerkmeister: installed beautifulsoup4 * 20:05 lucaswerkmeister: deployed {{Gerrit|ec15439d88}} (add missing CSRF checks 😱) * 16:56 lucaswerkmeister: briefly deployed and then undeployed again some experimental code to track slow queries (now in a slowqueries branch) * 14:02 lucaswerkmeister: deployed {{Gerrit|5e8877a1f4}} (link to batches list from index) * 13:58 lucaswerkmeister: deployed {{Gerrit|58a6f3e625}} (batches list) === 2019-05-03 === * 20:12 lucaswerkmeister: deployed {{Gerrit|7691176577}} (missing period) * 20:06 lucaswerkmeister: deployed {{Gerrit|3968d54ad9}} (batch summary) === 2019-05-02 === * 20:47 lucaswerkmeister: deployed {{Gerrit|d448b31405}} (show batch domain + title on index page) === 2019-05-01 === * 23:28 lucaswerkmeister: deployed {{Gerrit|13006c471b}} (cache titles/summaries) * 21:33 lucaswerkmeister: deployed {{Gerrit|7d2d1992bd}} (batch titles) * 12:28 lucaswerkmeister: deployed {{Gerrit|fdfaa14666}} * 11:32 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (123) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 11:25 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch IN (38, 40, 41, 42, 43, 44, 115, 128, 129, 170) AND command_status = 16; -- reset commands that for some reason never ran from pending to planned * 10:23 lucaswerkmeister: UPDATE command SET command_status = 0 WHERE command_batch = 236 AND command_status = 16; -- reset two commands that for some reason never ran from pending to planned === 2019-04-30 === * 23:10 lucaswerkmeister: deployed {{Gerrit|ce83ffb040}} === 2019-04-28 === * 20:49 lucaswerkmeister: deployed {{Gerrit|bd9b272b21}} * 16:14 lucaswerkmeister: deployed {{Gerrit|9d3c6e3dc3}} === 2019-04-22 === * 20:16 lucaswerkmeister: enable background runs for everyone * 20:02 lucaswerkmeister: deployed {{Gerrit|9c4941bc63}} === 2019-04-18 === * 19:56 lucaswerkmeister: deployed {{Gerrit|1d4e4fb16b}} === 2019-04-14 === * 12:23 lucaswerkmeister: kubectl delete pod quickcategories-654583560-xqip5 <noinclude>[[Category:SAL]]</noinclude> cedowta9ja34g8tuzc054mqi6sxyado 0 447389 2466270 2465967 2026-10-10T12:45:14Z LToscano (WMF) 7191 2466270 wikitext text/x-wiki Our PKI infrastructure is based on Cloudflare's [[github:cloudflare/cfssl|cfssl]], and in order to use it we maintain the following: * Puppet support to be able to provision and automatically renew a TLS certificate. * [[gerrit:admin/repos/operations/software/cfssl-issuer,general|cfssl-issuer]], an External Issuer for [[Kubernetes/cert-manager|cert-manager]] to be used with CFSSL <code>multirootca</code> and K8s (it is based off of the sample-external-issuer provided by cert-manager). * A [[gitlab:repos/sre/cfssl|fork]] of the cfssl repository to add [https://gitlab.wikimedia.org/repos/sre/cfssl/-/tree/main/debian/patches?ref_type=heads custom patches] that we filed to upstream but that were never merged. At the infrastructure level we run (Oct 2026): * One pki-root host in eqiad that hosts the cfssl root's ca. It is backed-up regularly via Bacula. * Multiple pki[12]* nodes running the cfssl-multiroot daemon. They are a mixture of VMs and bare metal hosts behind a LVS service. The idea is to avoid using only VMs due to the fact that Ganeti depends on PKI for TLS certs. * A Mariad db database supporting cfssl-multiroot daemons. It holds all certificates emitted, so that cfssl won't regenerate them if not needed. The last time that we upgraded PKI to a new version was tracked in https://phabricator.wikimedia.org/T416664. Security note: we rely on HMAC signing, provided/required by cfssl, as way to authenticate clients requesting a TLS certificate. Every group of clients (like every k8s cluster, puppet on bare metal, etc.) shares a secret password with cfssl, that allows to sign requests and authenticate them. == Useful links == {{Special:PrefixIndex/PKI/|hideredirects=1}} [[Category:SRE Infrastructure Foundations]] [[Category:PKI]] 80rnhmci3j99h83hvfgen6q0f3aqf8j 0 449160 2466278 2466260 2026-10-11T00:00:07Z Dexbot 30554 Bot: Updating the report 2466278 wikitext text/x-wiki {{/Header}} == Today (2026-10-11) == == Yesterday (2026-10-10) == == Last seven days == {| class="wikitable" |+ codfw |- ! Section !! Work |- | x1 || [[phab:T436922|Switchover x1 master (db2191 -&gt; db2249) (T436922)]] (marostegui) |- |} [[Category:MariaDB]] tncab4r6qcrkinqnshgbp1k54refp3r 116 453906 2466283 2466237 2026-10-11T04:27:16Z Gitlabaccountapprovalbot 37332 europepetrol was rejected. 2466283 wikitext text/x-wiki <noinclude>'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__</noinclude> === 2026-10-11 === * 04:27 "europepetrol" was rejected (pending since 2026-07-12T04:24:46.926Z). === 2026-10-09 === * 18:33 [[gitlab:stjn|@stjn]] was approved. * 14:39 "adarsh2406" was rejected (pending since 2026-07-10T14:39:14.167Z). === 2026-10-08 === * 09:51 [[gitlab:mathieudenelwmfr|@mathieudenelwmfr]] was approved. * 00:00 "mcint" was rejected (pending since 2026-07-08T23:58:27.716Z). === 2026-10-07 === * 20:51 [[gitlab:connieatwork|@connieatwork]] was approved. * 07:39 [[gitlab:peterxy12|@peterxy12]] was approved. * 05:15 "chuiimuiiofc" was rejected (pending since 2026-07-08T05:12:33.360Z). === 2026-10-06 === * 21:21 "giantflightlessbirds" was rejected (pending since 2026-07-07T21:20:24.197Z). === 2026-10-05 === * 04:18 "manjurul" was rejected (pending since 2026-07-06T04:16:41.067Z). === 2026-10-04 === * 09:57 "gdh-euwk" was rejected (pending since 2026-07-05T09:55:07.048Z). === 2026-10-02 === * 14:51 [[gitlab:nichesports|@nichesports]] was approved. * 12:57 [[gitlab:harii|@harii]] was approved. * 12:12 "mgagat" was rejected (pending since 2026-07-03T12:10:46.225Z). === 2026-09-29 === * 13:45 "monathierse" was rejected (pending since 2026-06-30T13:42:41.627Z). * 00:24 [[gitlab:balance3641|@balance3641]] was approved. === 2026-09-27 === * 20:12 [[gitlab:brokensegue|@brokensegue]] was approved. * 04:30 "bharu-123" was rejected (pending since 2026-06-28T04:29:14.159Z). === 2026-09-23 === * 17:48 "mdh" was rejected (pending since 2026-06-24T17:46:51.726Z). === 2026-09-19 === * 06:42 "thet1nk3r3r" was rejected (pending since 2026-06-20T06:40:39.066Z). === 2026-09-15 === * 19:57 "modalya" was rejected (pending since 2026-06-16T19:55:12.189Z). === 2026-09-14 === * 01:12 [[gitlab:11wb|@11wb]] was approved. === 2026-09-13 === * 20:54 [[gitlab:mrstarfleetcommand|@mrstarfleetcommand]] was approved. * 11:36 "nyxira17" was rejected (pending since 2026-06-14T11:35:22.416Z). === 2026-09-12 === * 19:06 "deuces9ers" was rejected (pending since 2026-06-13T19:04:55.275Z). === 2026-09-10 === * 08:54 "nehap1706" was rejected (pending since 2026-06-11T08:53:58.819Z). * 01:48 "tehkittycat" was rejected (pending since 2026-06-11T01:47:45.480Z). === 2026-09-09 === * 20:12 "lufi2586" was rejected (pending since 2026-06-10T20:09:59.034Z). === 2026-09-08 === * 02:39 "tantomile" was rejected (pending since 2026-06-09T02:36:22.460Z). === 2026-09-07 === * 15:45 [[gitlab:dvrted|@dvrted]] was approved. * 03:00 "jessica22664" was rejected (pending since 2026-06-08T02:58:30.150Z). === 2026-09-06 === * 21:36 "rp238" was rejected (pending since 2026-06-07T21:33:48.927Z). * 11:57 "versatiledove231" was rejected (pending since 2026-06-07T11:55:14.557Z). === 2026-09-05 === * 19:51 "paper9oll" was rejected (pending since 2026-06-06T19:48:46.353Z). * 04:21 "luminousstaring" was rejected (pending since 2026-06-06T04:20:44.385Z). === 2026-09-03 === * 08:09 [[gitlab:real68er|@real68er]] was approved. === 2026-09-02 === * 23:30 [[gitlab:aplucas0703|@aplucas0703]] was approved. * 16:42 "maximocampo" was rejected (pending since 2026-06-03T16:41:52.538Z). === 2026-09-01 === * 20:15 "sinaider2000" was rejected (pending since 2026-06-02T20:12:40.894Z). * 19:03 "t-rt15" was rejected (pending since 2026-06-02T19:02:56.111Z). * 04:21 "naomi723" was rejected (pending since 2026-06-02T04:20:21.044Z). === 2026-08-31 === * 20:06 "mccio" was rejected (pending since 2026-06-01T20:05:09.900Z). * 10:51 [[gitlab:hgzh3|@hgzh3]] was approved. === 2026-08-27 === * 02:18 [[gitlab:madmax|@madmax]] was approved. === 2026-08-26 === * 21:42 [[gitlab:mmns21|@mmns21]] was approved. === 2026-08-25 === * 17:27 [[gitlab:skim|@skim]] was approved. * 14:57 [[gitlab:wilfredor2|@wilfredor2]] was approved. * 08:03 [[gitlab:noahmvf|@noahmvf]] was approved. === 2026-08-23 === * 05:30 "amidewicki" was rejected (pending since 2026-05-24T05:29:29.216Z). * 01:54 "westcodragen07" was rejected (pending since 2026-05-24T01:53:53.396Z). === 2026-08-22 === * 15:21 [[gitlab:laraibhasan|@laraibhasan]] was approved. === 2026-08-20 === * 17:18 [[gitlab:jatwa|@jatwa]] was approved. === 2026-08-18 === * 11:33 "wladek92" was rejected (pending since 2026-05-19T11:30:32.679Z). === 2026-08-17 === * 18:39 [[gitlab:erlanger16e|@erlanger16e]] was approved. * 18:33 "ashishkaranam" was rejected (pending since 2026-05-18T18:32:18.813Z). * 12:48 [[gitlab:lfs|@lfs]] was approved. === 2026-08-16 === * 15:42 [[gitlab:pcendrer|@pcendrer]] was approved. === 2026-08-15 === * 17:33 [[gitlab:flammablepizza|@flammablepizza]] was approved. === 2026-08-14 === * 13:00 [[gitlab:pacmand|@pacmand]] was approved. === 2026-08-12 === * 13:21 [[gitlab:doublechek|@doublechek]] was approved. === 2026-08-11 === * 14:15 [[gitlab:mmilenkovicwmf|@mmilenkovicwmf]] was approved. === 2026-08-10 === * 17:18 [[gitlab:mustafe|@mustafe]] was approved. * 10:33 "c8263a20" was rejected (pending since 2026-05-11T10:32:40.353Z). === 2026-08-09 === * 21:18 [[gitlab:iamnetx|@iamnetx]] was approved. * 20:09 "yirba" was rejected (pending since 2026-05-10T20:07:07.738Z). * 06:42 "marsam2489" was rejected (pending since 2026-05-10T06:40:46.276Z). === 2026-08-08 === * 08:15 [[gitlab:taiwaniajusto|@taiwaniajusto]] was approved. === 2026-08-07 === * 18:21 [[gitlab:gturkington|@gturkington]] was approved. * 06:36 "brianbybyby" was rejected (pending since 2026-05-08T06:36:08.459Z). === 2026-07-30 === * 20:42 "horaciocolbert" was rejected (pending since 2026-04-30T20:41:40.421Z). === 2026-07-29 === * 10:03 [[gitlab:piastu|@piastu]] was approved. * 06:33 "rafiul1" was rejected (pending since 2026-04-29T06:33:02.573Z). === 2026-07-28 === * 16:51 [[gitlab:for-each-next|@for-each-next]] was approved. * 09:21 [[gitlab:gka|@gka]] was approved. === 2026-07-27 === * 08:12 [[gitlab:cambob|@cambob]] was approved. === 2026-07-26 === * 15:39 "demansanaagmailcom" was rejected (pending since 2026-04-26T15:39:03.297Z). === 2026-07-24 === * 10:54 [[gitlab:ysogo|@ysogo]] was approved. * 09:39 [[gitlab:pankaj199|@pankaj199]] was approved. === 2026-07-23 === * 12:30 [[gitlab:fermiboson|@fermiboson]] was approved. * 09:21 [[gitlab:slashme|@slashme]] was approved. === 2026-07-22 === * 15:54 [[gitlab:lmedley|@lmedley]] was approved. * 13:24 "praveen5638" was rejected (pending since 2026-04-22T13:21:23.368Z). * 12:48 [[gitlab:cyberpower678|@cyberpower678]] was approved. * 08:30 [[gitlab:nabbegat|@nabbegat]] was approved. * 08:30 [[gitlab:plyd|@plyd]] was approved. * 07:06 "ayush8620" was rejected (pending since 2026-04-22T07:03:12.476Z). === 2026-07-21 === * 15:51 [[gitlab:panieravide|@panieravide]] was approved. * 15:33 [[gitlab:luisvilla-personal|@luisvilla-personal]] was approved. * 13:48 [[gitlab:yru|@yru]] was approved. * 13:09 [[gitlab:deevad|@deevad]] was approved. * 12:54 [[gitlab:ctdo17|@ctdo17]] was approved. * 12:36 [[gitlab:jeannenoiraud|@jeannenoiraud]] was approved. * 12:21 [[gitlab:nadiantara|@nadiantara]] was approved. * 12:18 [[gitlab:wijltcher|@wijltcher]] was approved. * 10:33 [[gitlab:nivopol|@nivopol]] was approved. * 10:30 [[gitlab:johlig|@johlig]] was approved. * 10:30 [[gitlab:majicita|@majicita]] was approved. * 10:15 [[gitlab:yongjiapeng|@yongjiapeng]] was approved. * 10:09 [[gitlab:francyskus|@francyskus]] was approved. * 09:24 [[gitlab:xanonymusx|@xanonymusx]] was approved. === 2026-07-20 === * 17:15 "leonidlednev" was rejected (pending since 2026-04-20T17:13:35.108Z). * 15:27 [[gitlab:rodrigoargenton|@rodrigoargenton]] was approved. * 05:48 "draftecho" was rejected (pending since 2026-04-20T05:48:06.953Z). === 2026-07-19 === * 12:21 [[gitlab:boivie|@boivie]] was approved. === 2026-07-18 === * 16:09 [[gitlab:pharos|@pharos]] was approved. * 15:45 [[gitlab:priyankar22|@priyankar22]] was approved. * 15:30 [[gitlab:sisyph|@sisyph]] was approved. === 2026-07-13 === * 03:45 [[gitlab:dreamyshade|@dreamyshade]] was approved. === 2026-07-12 === * 09:27 [[gitlab:smk|@smk]] was approved. === 2026-07-11 === * 14:48 "bigcereal42" was rejected (pending since 2026-04-11T14:47:40.321Z). * 12:18 "pratyushsawan" was rejected (pending since 2026-04-11T12:16:41.671Z). === 2026-07-10 === * 14:42 [[gitlab:akaza24|@akaza24]] was approved. * 12:57 [[gitlab:kormisk|@kormisk]] was approved. === 2026-07-07 === * 10:36 [[gitlab:olafjanssen|@olafjanssen]] was approved. * 06:57 "elisapoly-99" was rejected (pending since 2026-04-07T06:55:52.662Z). === 2026-07-06 === * 11:57 "ma3rouf" was rejected (pending since 2026-04-06T11:56:30.978Z). === 2026-07-02 === * 15:48 [[gitlab:tekneos|@tekneos]] was approved. === 2026-07-01 === * 15:12 [[gitlab:mugurolevy|@mugurolevy]] was approved. * 14:15 [[gitlab:vadymts1|@vadymts1]] was approved. * 09:57 "mugurolevy" was rejected (pending since 2026-04-01T09:55:19.175Z). === 2026-06-30 === * 14:27 "shivangisharma" was rejected (pending since 2026-03-31T14:26:44.932Z). === 2026-06-29 === * 19:03 [[gitlab:thisismattmiller|@thisismattmiller]] was approved. === 2026-06-28 === * 14:51 "nkwenuinadine" was rejected (pending since 2026-03-29T14:48:32.735Z). * 14:03 "vaishnavikumbhar" was rejected (pending since 2026-03-29T14:01:30.604Z). * 13:03 "stepmay" was rejected (pending since 2026-03-29T13:01:59.905Z). * 06:51 "swallroth" was rejected (pending since 2026-03-29T06:49:54.838Z). === 2026-06-26 === * 09:39 [[gitlab:lakshita28|@lakshita28]] was approved. * 07:30 [[gitlab:reeti|@reeti]] was approved. * 07:30 [[gitlab:anushka10patel|@anushka10patel]] was approved. * 07:30 "samsaesque" was rejected (pending since 2026-03-27T07:29:57.279Z). * 05:51 [[gitlab:arpithhhaaa|@arpithhhaaa]] was approved. * 05:51 [[gitlab:govindlaltl|@govindlaltl]] was approved. === 2026-06-25 === * 16:09 [[gitlab:sakuraemad|@sakuraemad]] was approved. * 07:00 "kdh8219" was rejected (pending since 2026-03-26T06:58:05.415Z). === 2026-06-24 === * 11:54 [[gitlab:sanskardubeydev|@sanskardubeydev]] was approved. * 10:09 "tanmay789q" was rejected (pending since 2026-03-25T10:07:54.602Z). === 2026-06-22 === * 19:57 [[gitlab:gouvernathor|@gouvernathor]] was approved. * 16:45 [[gitlab:lucasbelo|@lucasbelo]] was approved. * 07:15 "jason2000-cpu" was rejected (pending since 2026-03-23T07:14:09.184Z). === 2026-06-21 === * 13:18 [[gitlab:egonw|@egonw]] was approved. === 2026-06-20 === * 10:21 [[gitlab:tways2017|@tways2017]] was approved. === 2026-06-19 === * 16:06 "wilsonwang2026" was rejected (pending since 2026-03-20T16:06:05.511Z). * 04:12 [[gitlab:claudio|@claudio]] was approved. === 2026-06-18 === * 14:21 "royiswariii" was rejected (pending since 2026-03-19T14:19:16.896Z). * 13:06 [[gitlab:laurabarluzzi|@laurabarluzzi]] was approved. === 2026-06-17 === * 11:24 "adinathq8x" was rejected (pending since 2026-03-18T11:22:50.098Z). * 09:45 "nathanveritas" was rejected (pending since 2026-03-18T09:43:51.645Z). === 2026-06-15 === * 22:39 [[gitlab:mohammadhijjawi|@mohammadhijjawi]] was approved. * 14:24 "enlisar" was rejected (pending since 2026-03-16T14:23:00.109Z). * 14:06 "ayaan" was rejected (pending since 2026-03-16T14:03:31.071Z). * 10:54 "kwametech" was rejected (pending since 2026-03-16T10:54:11.083Z). === 2026-06-14 === * 17:45 [[gitlab:surajseth520|@surajseth520]] was approved. * 07:24 "malahimhaseeb" was rejected (pending since 2026-03-15T07:21:57.748Z). === 2026-06-11 === * 11:48 [[gitlab:cadddr|@cadddr]] was approved. * 11:18 "wikipiggy" was rejected (pending since 2026-03-12T11:16:09.335Z). * 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved. === 2026-06-10 === * 07:03 [[gitlab:dmiranda|@dmiranda]] was approved. === 2026-06-09 === * 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved. * 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved. * 12:51 [[gitlab:ekrem|@ekrem]] was approved. === 2026-06-08 === * 17:48 "jmprax" was rejected (pending since 2026-03-09T17:46:38.807Z). * 16:15 [[gitlab:ahonc|@ahonc]] was approved. * 12:57 [[gitlab:rainmonger|@rainmonger]] was approved. === 2026-06-07 === * 23:03 "shadowthewuff" was rejected (pending since 2026-03-08T23:00:53.442Z). * 11:45 "wiki-pavan" was rejected (pending since 2026-03-08T11:45:11.116Z). * 02:39 [[gitlab:launchpad|@launchpad]] was approved. === 2026-06-06 === * 14:54 "unicord" was rejected (pending since 2026-03-07T14:52:04.992Z). * 12:48 "chien" was rejected (pending since 2026-03-07T12:48:11.669Z). === 2026-06-04 === * 14:33 "only-vikas" was rejected (pending since 2026-03-05T14:32:09.186Z). === 2026-06-03 === * 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved. === 2026-06-02 === * 21:21 "mgagat" was rejected (pending since 2026-03-03T21:18:37.223Z). * 13:57 "prasunaenumarthy" was rejected (pending since 2026-03-03T13:57:14.847Z). * 05:48 [[gitlab:tmoney|@tmoney]] was approved. === 2026-06-01 === * 14:57 "vikram2101" was rejected (pending since 2026-03-02T14:54:26.550Z). * 12:03 "watshell" was rejected (pending since 2026-03-02T12:03:09.329Z). === 2026-05-29 === * 12:48 "mounikapotladurthi" was rejected (pending since 2026-02-27T12:45:38.609Z). === 2026-05-27 === * 20:00 "vinitha" was rejected (pending since 2026-02-25T19:58:43.524Z). * 16:30 "codeurluce" was rejected (pending since 2026-02-25T16:28:53.973Z). * 14:33 [[gitlab:thilio|@thilio]] was approved. === 2026-05-26 === * 12:09 "charisad" was rejected (pending since 2026-02-24T12:07:21.881Z). === 2026-05-25 === * 22:54 "ddshelto" was rejected (pending since 2026-02-23T22:52:44.427Z). * 19:51 "lakz-99" was rejected (pending since 2026-02-23T19:47:00.263Z). * 19:48 "lakz-99" was rejected (pending since 2026-02-23T19:47:00.263Z). === 2026-05-24 === * 18:45 "jiyagupta-cs" was rejected (pending since 2026-02-22T18:43:33.176Z). === 2026-05-23 === * 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved. * 04:21 [[gitlab:staraction|@staraction]] was approved. === 2026-05-22 === * 19:03 "i-horich" was rejected (pending since 2026-02-20T19:00:43.519Z). * 01:48 "50323233" was rejected (pending since 2026-02-20T01:48:05.555Z). === 2026-05-21 === * 18:51 "kartikeyg0104" was rejected (pending since 2026-02-19T18:48:39.707Z). * 16:27 [[gitlab:renovatebot|@renovatebot]] was approved. * 16:06 [[gitlab:gkm563|@gkm563]] was approved. === 2026-05-20 === * 01:21 "beedellrokejulianlockhart" was rejected (pending since 2026-02-18T01:19:13.284Z). === 2026-05-18 === * 23:18 "wladek92" was rejected (pending since 2026-02-16T23:16:22.939Z). * 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved. === 2026-05-14 === * 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved. === 2026-05-13 === * 10:51 "ssssaaaa" was rejected (pending since 2026-02-11T10:50:36.975Z). === 2026-05-12 === * 18:06 [[gitlab:psubhashish|@psubhashish]] was approved. * 08:12 "khan" was rejected (pending since 2026-02-10T08:11:48.776Z). * 04:27 "galaxysh" was rejected (pending since 2026-02-10T04:24:59.440Z). === 2026-05-11 === * 12:18 "peterxy12" was rejected (pending since 2026-02-09T12:18:01.982Z). === 2026-05-10 === * 11:09 "yalihupokn" was rejected (pending since 2026-02-08T11:06:51.336Z). * 05:12 "wobadha" was rejected (pending since 2026-02-08T05:11:00.569Z). === 2026-05-09 === * 13:45 "bwiki" was rejected (pending since 2026-02-07T13:43:38.177Z). === 2026-05-08 === * 09:24 [[gitlab:cwilliams|@cwilliams]] was approved. === 2026-05-07 === * 14:15 "rehankhan78" was rejected (pending since 2026-02-05T14:13:37.754Z). === 2026-05-06 === * 11:24 "ari" was rejected (pending since 2026-02-04T11:24:11.760Z). * 08:09 [[gitlab:neriah|@neriah]] was approved. * 06:27 [[gitlab:status401|@status401]] was approved. === 2026-05-03 === * 09:54 [[gitlab:anilk|@anilk]] was approved. === 2026-05-02 === * 17:54 [[gitlab:sweil|@sweil]] was approved. * 17:00 [[gitlab:aoppo|@aoppo]] was approved. === 2026-05-01 === * 21:18 [[gitlab:dawalda|@dawalda]] was approved. === 2026-04-30 === * 21:42 "merohibine" was rejected (pending since 2026-01-29T21:40:00.756Z). * 20:54 [[gitlab:tfmorris|@tfmorris]] was approved. * 17:33 [[gitlab:uyen|@uyen]] was approved. * 07:39 [[gitlab:mahveotm|@mahveotm]] was approved. * 06:36 [[gitlab:leo321|@leo321]] was approved. === 2026-04-29 === * 02:27 [[gitlab:dw31415|@dw31415]] was approved. === 2026-04-28 === * 23:09 [[gitlab:dtorsani|@dtorsani]] was approved. === 2026-04-27 === * 23:42 [[gitlab:quinlan|@quinlan]] was approved. * 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved. === 2026-04-26 === * 17:36 "kuba-hajnej" was rejected (pending since 2026-01-25T17:33:32.467Z). * 13:03 "jklamo" was rejected (pending since 2026-01-25T13:02:22.936Z). === 2026-04-25 === * 20:24 [[gitlab:maldaxura|@maldaxura]] was approved. * 14:33 [[gitlab:sirtobi|@sirtobi]] was approved. * 04:18 "ice5678" was rejected (pending since 2026-01-24T04:15:30.008Z). === 2026-04-24 === * 22:06 [[gitlab:arcstur|@arcstur]] was approved. === 2026-04-22 === * 23:06 "dtorsani" was rejected (pending since 2026-01-21T23:03:25.843Z). * 22:18 [[gitlab:egezort|@egezort]] was approved. * 16:45 "nexpectarpit" was rejected (pending since 2026-01-21T16:43:21.045Z). === 2026-04-20 === * 19:15 "fitch" was rejected (pending since 2026-01-19T19:12:35.644Z). === 2026-04-19 === * 02:54 [[gitlab:neoact|@neoact]] was approved. === 2026-04-18 === * 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved. === 2026-04-17 === * 13:42 "liselot" was rejected (pending since 2026-01-16T13:39:41.909Z). === 2026-04-15 === * 17:03 "lahari" was rejected (pending since 2026-01-14T17:02:06.275Z). === 2026-04-14 === * 13:00 "surajseth520" was rejected (pending since 2026-01-13T12:59:45.906Z). * 04:51 [[gitlab:canley|@canley]] was approved. * 01:03 "bshizzle" was rejected (pending since 2026-01-13T01:00:48.120Z). === 2026-04-13 === * 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved. === 2026-04-11 === * 12:30 "krithash" was rejected (pending since 2026-01-10T12:27:24.731Z). === 2026-04-10 === * 15:30 "raunak1709" was rejected (pending since 2026-01-09T15:29:10.901Z). === 2026-04-07 === * 17:03 [[gitlab:supnabla|@supnabla]] was approved. === 2026-04-06 === * 20:00 [[gitlab:laerdon|@laerdon]] was approved. * 19:21 [[gitlab:ljq3|@ljq3]] was approved. === 2026-04-04 === * 11:06 "mixcc" was rejected (pending since 2026-01-03T11:03:33.922Z). === 2026-04-02 === * 05:30 [[gitlab:mbh1|@mbh1]] was approved. === 2026-04-01 === * 18:21 "yuvrajpatil17" was rejected (pending since 2025-12-31T18:20:27.991Z). * 12:12 [[gitlab:amorii0|@amorii0]] was approved. === 2026-03-31 === * 11:00 "krrishsehgal" was rejected (pending since 2025-12-30T11:00:16.384Z). === 2026-03-30 === * 15:36 [[gitlab:atsuko|@atsuko]] was approved. === 2026-03-29 === * 11:36 [[gitlab:giftcup|@giftcup]] was approved. === 2026-03-28 === * 14:51 [[gitlab:janeeva1|@janeeva1]] was approved. === 2026-03-26 === * 13:36 [[gitlab:saiphani02|@saiphani02]] was approved. * 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved. === 2026-03-25 === * 09:45 "quansi" was rejected (pending since 2025-12-24T09:42:13.451Z). * 02:18 [[gitlab:viztor|@viztor]] was approved. === 2026-03-24 === * 23:18 [[gitlab:maryyann|@maryyann]] was approved. * 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved. * 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved. * 07:39 "fred2675" was rejected (pending since 2025-12-23T07:39:11.380Z). === 2026-03-23 === * 14:51 [[gitlab:komla|@komla]] was approved. * 05:51 "lunachuck43" was rejected (pending since 2025-12-22T05:50:17.862Z). * 04:06 "reza110011" was rejected (pending since 2025-12-22T04:05:25.117Z). === 2026-03-20 === * 21:54 "mertgor" was rejected (pending since 2025-12-19T21:51:51.419Z). * 20:57 "autanmahmah" was rejected (pending since 2025-12-19T20:54:51.678Z). * 09:57 [[gitlab:nethahussain|@nethahussain]] was approved. * 09:27 [[gitlab:piewriter|@piewriter]] was approved. * 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved. === 2026-03-19 === * 21:03 "sayvhior" was rejected (pending since 2025-12-18T21:02:31.699Z). === 2026-03-18 === * 20:15 [[gitlab:martinmystere|@martinmystere]] was approved. === 2026-03-17 === * 02:51 "louperivois" was rejected (pending since 2025-12-16T02:50:48.197Z). === 2026-03-16 === * 12:54 "mokayaj857" was rejected (pending since 2025-12-15T12:53:39.015Z). * 06:18 "roamer15" was rejected (pending since 2025-12-15T06:16:38.042Z). === 2026-03-14 === * 11:12 "umaramuhammad" was rejected (pending since 2025-12-13T11:10:44.004Z). * 09:33 "akuma19" was rejected (pending since 2025-12-13T09:31:39.044Z). * 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved. === 2026-03-12 === * 20:24 [[gitlab:11wb|@11wb]] was approved. * 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved. === 2026-03-10 === * 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved. === 2026-03-06 === * 08:09 "vazhayilnewone" was rejected (pending since 2025-12-05T08:07:02.184Z). === 2026-03-04 === * 20:54 [[gitlab:elphie|@elphie]] was approved. * 11:39 "ronaldahmed" was rejected (pending since 2025-12-03T11:37:47.492Z). * 02:12 "ltslw" was rejected (pending since 2025-12-03T02:11:52.040Z). === 2026-03-02 === * 19:21 "dlopez350" was rejected (pending since 2025-12-01T19:20:38.918Z). * 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved. === 2026-03-01 === * 10:51 [[gitlab:clintacc|@clintacc]] was approved. === 2026-02-28 === * 09:24 "cardboardlamp" was rejected (pending since 2025-11-29T09:22:03.947Z). * 08:18 "wiki-pavan" was rejected (pending since 2025-11-29T08:16:24.184Z). === 2026-02-27 === * 20:45 "thisisrick25" was rejected (pending since 2025-11-28T20:42:24.454Z). === 2026-02-26 === * 13:57 "chuiimuiiofc" was rejected (pending since 2025-11-27T13:57:02.794Z). * 13:54 "steffpro" was rejected (pending since 2025-11-27T13:52:10.859Z). === 2026-02-25 === * 21:24 "abubakarhabibudayyabu" was rejected (pending since 2025-11-26T21:22:37.776Z). === 2026-02-24 === * 05:00 "playboi" was rejected (pending since 2025-11-25T05:00:30.762Z). === 2026-02-23 === * 14:00 "alph65" was rejected (pending since 2025-11-24T13:59:00.797Z). * 12:33 [[gitlab:robertsky|@robertsky]] was approved. === 2026-02-22 === * 00:30 "hp8p" was rejected (pending since 2025-11-23T00:29:24.741Z). === 2026-02-19 === * 16:45 "clayjar" was rejected (pending since 2025-11-20T16:44:48.380Z). === 2026-02-18 === * 22:18 "nexus" was rejected (pending since 2025-11-19T22:16:48.818Z). * 12:00 "bernsteinnn" was rejected (pending since 2025-11-19T11:59:04.427Z). === 2026-02-17 === * 11:36 "jason2000-cpu" was rejected (pending since 2025-11-18T11:34:00.314Z). === 2026-02-16 === * 14:54 "smaurya" was rejected (pending since 2025-11-17T14:52:06.906Z). === 2026-02-15 === * 16:51 "kra-79" was rejected (pending since 2025-11-16T16:50:41.375Z). === 2026-02-14 === * 15:15 [[gitlab:mess|@mess]] was approved. === 2026-02-13 === * 13:57 "sopalsuemae957" was rejected (pending since 2025-11-14T13:55:16.921Z). * 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved. === 2026-02-12 === * 16:30 "kristinagligoric" was rejected (pending since 2025-11-13T16:29:21.646Z). * 03:33 [[gitlab:anyehansen|@anyehansen]] was approved. * 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved. === 2026-02-10 === * 13:18 [[gitlab:db111|@db111]] was approved. === 2026-02-09 === * 19:06 "squirrel289" was rejected (pending since 2025-11-10T19:04:27.831Z). === 2026-02-06 === * 20:54 [[gitlab:gillux|@gillux]] was approved. * 09:09 [[gitlab:lih|@lih]] was approved. === 2026-01-31 === * 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved. === 2026-01-28 === * 14:30 [[gitlab:ademola|@ademola]] was approved. * 10:51 "watshell" was rejected (pending since 2025-10-29T10:51:01.521Z). === 2026-01-26 === * 23:06 "tavaresgmg" was rejected (pending since 2025-10-27T23:04:42.140Z). === 2026-01-25 === * 06:03 "cata" was rejected (pending since 2025-10-26T06:01:26.155Z). === 2026-01-24 === * 21:15 [[gitlab:wiegels|@wiegels]] was approved. * 06:30 [[gitlab:blaquans|@blaquans]] was approved. === 2026-01-23 === * 16:27 [[gitlab:lerickson|@lerickson]] was approved. * 10:15 "fran0035g" was rejected (pending since 2025-10-24T10:12:17.732Z). === 2026-01-22 === * 21:00 "hacksyn" was rejected (pending since 2025-10-23T20:59:15.982Z). === 2026-01-21 === * 17:30 [[gitlab:otcenas11|@otcenas11]] was approved. === 2026-01-19 === * 21:48 [[gitlab:amdrel|@amdrel]] was approved. * 04:36 "rayalexa" was rejected (pending since 2025-10-20T04:35:02.094Z). === 2026-01-18 === * 15:45 "somya" was rejected (pending since 2025-10-19T15:43:43.701Z). * 06:54 "sergg001" was rejected (pending since 2025-10-19T06:54:12.296Z). === 2026-01-16 === * 11:57 "zeejohsy" was rejected (pending since 2025-10-17T11:56:22.372Z). * 04:45 "rocky25" was rejected (pending since 2025-10-17T04:43:33.180Z). === 2026-01-15 === * 16:39 "tiisu" was rejected (pending since 2025-10-16T16:37:18.438Z). * 12:00 "noahalorwu" was rejected (pending since 2025-10-16T11:58:26.133Z). * 10:39 "prjayaiuedu" was rejected (pending since 2025-10-16T10:37:16.947Z). === 2026-01-13 === * 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved. === 2026-01-12 === * 17:03 "stagietechs" was rejected (pending since 2025-10-13T17:02:25.281Z). === 2026-01-10 === * 19:06 "keerthisr" was rejected (pending since 2025-10-11T19:05:01.758Z). === 2026-01-09 === * 20:36 "lightb" was rejected (pending since 2025-10-10T20:34:20.264Z). === 2026-01-08 === * 19:42 [[gitlab:tbodt|@tbodt]] was approved. * 13:57 [[gitlab:martynranyard|@martynranyard]] was approved. === 2026-01-07 === * 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved. * 14:27 "dipanshu" was rejected (pending since 2025-10-08T14:26:10.794Z). * 12:30 "adeolaadesina" was rejected (pending since 2025-10-08T12:29:49.592Z). * 09:21 "tony-kamande" was rejected (pending since 2025-10-08T09:20:28.421Z). * 06:18 "hninwuttyi" was rejected (pending since 2025-10-08T06:17:28.006Z). * 05:09 "andume" was rejected (pending since 2025-10-08T05:07:18.582Z). * 02:00 "mosope" was rejected (pending since 2025-10-08T01:59:54.800Z). * 01:15 [[gitlab:tungstalite|@tungstalite]] was approved. === 2026-01-06 === * 18:24 "leerensucher" was rejected (pending since 2025-10-07T18:21:41.253Z). * 14:54 "leonidlednev" was rejected (pending since 2025-10-07T14:53:07.273Z). * 12:57 "alexandre-tingaud" was rejected (pending since 2025-10-07T12:54:27.206Z). === 2026-01-04 === * 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved. * 15:18 "makjr" was rejected (pending since 2025-10-05T15:16:31.558Z). * 14:09 "dakshq" was rejected (pending since 2025-10-05T14:08:40.608Z). === 2026-01-03 === * 20:42 [[gitlab:apehitkey|@apehitkey]] was approved. * 18:00 [[gitlab:jeremyb|@jeremyb]] was approved. * 14:09 [[gitlab:twelephant|@twelephant]] was approved. === 2026-01-01 === * 11:30 "shellstanislav" was rejected (pending since 2025-10-02T11:29:10.150Z). === 2025-12-30 === * 19:51 "camilojdiaz" was rejected (pending since 2025-09-30T19:49:24.913Z). === 2025-12-29 === * 16:03 "zied" was rejected (pending since 2025-09-29T16:01:30.415Z). * 08:18 "rahulsidpradhan" was rejected (pending since 2025-09-29T08:17:02.849Z). === 2025-12-26 === * 09:48 "thembo42" was rejected (pending since 2025-09-26T09:45:15.033Z). === 2025-12-25 === * 14:03 "196936074751" was rejected (pending since 2025-09-25T14:02:31.367Z). === 2025-12-23 === * 16:21 "ngarnsworthy" was rejected (pending since 2025-09-23T16:20:41.211Z). === 2025-12-22 === * 12:39 "aza555" was rejected (pending since 2025-09-22T12:38:02.622Z). === 2025-12-20 === * 23:45 "saph" was rejected (pending since 2025-09-20T23:45:01.222Z). === 2025-12-19 === * 10:15 "vladdymoses" was rejected (pending since 2025-09-19T10:15:00.999Z). * 07:15 "dirtylittlepoobah" was rejected (pending since 2025-09-19T07:13:55.537Z). === 2025-12-18 === * 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved. === 2025-12-17 === * 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved. * 18:30 "prudencia" was rejected (pending since 2025-09-17T18:27:18.860Z). * 02:24 "lottie" was rejected (pending since 2025-09-17T02:21:21.744Z). === 2025-12-16 === * 09:39 [[gitlab:melcatherine|@melcatherine]] was approved. * 08:54 [[gitlab:leila237|@leila237]] was approved. === 2025-12-15 === * 18:27 [[gitlab:royalsailor|@royalsailor]] was approved. * 09:39 [[gitlab:olaf8940|@olaf8940]] was approved. * 09:39 "brianbybyby" was rejected (pending since 2025-09-15T09:37:45.430Z). === 2025-12-14 === * 20:21 [[gitlab:essa237|@essa237]] was approved. * 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved. === 2025-12-13 === * 21:54 "mmns21" was rejected (pending since 2025-09-13T21:52:24.017Z). * 20:33 "bugcrawler" was rejected (pending since 2025-09-13T20:31:09.211Z). === 2025-12-12 === * 14:39 "ruvchoudhary" was rejected (pending since 2025-09-12T14:36:16.167Z). * 06:54 "rezadress" was rejected (pending since 2025-09-12T06:52:21.749Z). === 2025-12-10 === * 17:30 [[gitlab:itsmoon|@itsmoon]] was approved. === 2025-12-09 === * 15:42 [[gitlab:mercy-o|@mercy-o]] was approved. === 2025-12-06 === * 16:45 "jacquesradjabu" was rejected (pending since 2025-09-06T16:45:17.969Z). * 11:27 [[gitlab:ikhitron|@ikhitron]] was approved. === 2025-12-01 === * 08:12 "halconmilenario21" was rejected (pending since 2025-09-01T08:12:10.262Z). === 2025-11-30 === * 21:06 [[gitlab:habs|@habs]] was approved. === 2025-11-29 === * 16:36 "bovimacoco" was rejected (pending since 2025-08-30T16:34:39.712Z). * 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved. === 2025-11-24 === * 10:30 "alph65" was rejected (pending since 2025-08-25T10:28:40.957Z). * 02:24 [[gitlab:yaron|@yaron]] was approved. === 2025-11-20 === * 16:06 "clayjar" was rejected (pending since 2025-08-21T16:04:54.450Z). === 2025-11-17 === * 21:09 [[gitlab:ankita97531|@ankita97531]] was approved. === 2025-11-16 === * 14:15 "commanderkefir" was rejected (pending since 2025-08-17T14:13:14.791Z). * 08:21 "rehankhan78" was rejected (pending since 2025-08-17T08:19:44.896Z). === 2025-11-15 === * 14:36 "cyberscribe" was rejected (pending since 2025-08-16T14:34:27.230Z). === 2025-11-13 === * 04:21 "waddie96" was rejected (pending since 2025-08-14T04:19:27.461Z). === 2025-11-11 === * 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved. === 2025-11-10 === * 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved. === 2025-11-09 === * 22:36 "heinxiety" was rejected (pending since 2025-08-10T22:33:12.041Z). === 2025-11-07 === * 22:00 [[gitlab:forzagreen|@forzagreen]] was approved. === 2025-11-06 === * 16:57 [[gitlab:rsilvola|@rsilvola]] was approved. === 2025-11-04 === * 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved. === 2025-11-03 === * 17:48 "joewaleed98" was rejected (pending since 2025-08-04T17:46:12.191Z). === 2025-11-01 === * 18:00 "eliasempresas" was rejected (pending since 2025-08-02T17:58:04.412Z). === 2025-10-31 === * 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved. * 04:33 "3ch310n" was rejected (pending since 2025-08-01T04:32:21.982Z). === 2025-10-30 === * 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved. === 2025-10-29 === * 14:54 "theap" was rejected (pending since 2025-07-30T14:52:12.066Z). === 2025-10-28 === * 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved. === 2025-10-27 === * 07:51 [[gitlab:jmoore111|@jmoore111]] was approved. === 2025-10-25 === * 21:09 [[gitlab:valor|@valor]] was approved. * 21:03 [[gitlab:booksmurf|@booksmurf]] was approved. * 02:48 "mystyc1" was rejected (pending since 2025-07-26T02:46:19.373Z). === 2025-10-24 === * 05:12 "aadarshmahesh" was rejected (pending since 2025-07-25T05:09:38.264Z). === 2025-10-22 === * 20:54 [[gitlab:janewanga|@janewanga]] was approved. * 17:27 "abeljeevan" was rejected (pending since 2025-07-23T17:26:46.884Z). * 16:12 "shrimpnaur" was rejected (pending since 2025-07-23T16:10:37.864Z). === 2025-10-21 === * 18:51 "jrmuizel" was rejected (pending since 2025-07-22T18:50:07.315Z). * 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved. === 2025-10-17 === * 13:21 [[gitlab:blegodwin|@blegodwin]] was approved. === 2025-10-16 === * 14:51 [[gitlab:bahago|@bahago]] was approved. * 14:12 "harikrishna0005" was rejected (pending since 2025-07-17T14:10:48.385Z). * 14:09 "gauthammohanraj" was rejected (pending since 2025-07-17T14:08:47.643Z). === 2025-10-15 === * 13:48 [[gitlab:adwivedii|@adwivedii]] was approved. * 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved. * 13:03 "childmnajennifer" was rejected (pending since 2025-07-16T13:01:50.236Z). * 05:06 "vssb4214" was rejected (pending since 2025-07-16T05:05:33.985Z). === 2025-10-14 === * 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved. * 15:33 [[gitlab:sadrettin|@sadrettin]] was approved. * 14:18 [[gitlab:tmwyk|@tmwyk]] was approved. * 08:42 "yasu0796" was rejected (pending since 2025-07-15T08:41:26.453Z). === 2025-10-13 === * 16:09 [[gitlab:atlas0007|@atlas0007]] was approved. === 2025-10-11 === * 17:42 [[gitlab:techwizzie|@techwizzie]] was approved. === 2025-10-10 === * 19:03 [[gitlab:miiswom|@miiswom]] was approved. * 16:06 [[gitlab:ninatakang|@ninatakang]] was approved. === 2025-10-09 === * 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved. * 14:21 [[gitlab:lebogang|@lebogang]] was approved. * 14:15 [[gitlab:kimondorose|@kimondorose]] was approved. * 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved. * 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved. * 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved. * 13:45 [[gitlab:system625|@system625]] was approved. * 13:45 [[gitlab:rolalove|@rolalove]] was approved. * 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved. * 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved. * 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved. * 13:36 [[gitlab:aradhya|@aradhya]] was approved. * 09:57 "bognd" was rejected (pending since 2025-07-10T09:55:48.661Z). === 2025-10-08 === * 23:36 [[gitlab:sopzy|@sopzy]] was approved. * 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved. * 19:39 [[gitlab:levon003|@levon003]] was approved. * 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved. * 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved. * 13:36 [[gitlab:chumzine|@chumzine]] was approved. * 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved. * 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved. * 03:42 "nazhiba" was rejected (pending since 2025-07-09T03:40:12.625Z). * 02:12 "mafennel" was rejected (pending since 2025-07-09T02:11:40.598Z). === 2025-10-07 === * 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved. * 21:30 [[gitlab:rona|@rona]] was approved. * 21:09 [[gitlab:sandijigs|@sandijigs]] was approved. * 13:36 "xisbajao" was rejected (pending since 2025-07-08T13:33:35.018Z). * 01:36 "areczek94" was rejected (pending since 2025-07-08T01:35:40.633Z). === 2025-10-06 === * 19:21 "wmcarter2017" was rejected (pending since 2025-07-07T19:21:12.899Z). === 2025-10-05 === * 14:15 "meetmendapara" was rejected (pending since 2025-07-06T14:14:16.726Z). === 2025-10-04 === * 20:51 "nftbaee" was rejected (pending since 2025-07-05T20:50:57.688Z). === 2025-10-03 === * 06:12 [[gitlab:javiermonton|@javiermonton]] was approved. === 2025-10-02 === * 20:15 "talaqalotaibipmp" was rejected (pending since 2025-07-03T20:13:05.164Z). === 2025-10-01 === * 10:54 "bjensen" was rejected (pending since 2025-07-02T10:53:46.574Z). * 02:45 "kowal1984" was rejected (pending since 2025-07-02T02:44:56.946Z). === 2025-09-30 === * 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved. * 00:24 "adium" was rejected (pending since 2025-07-01T00:23:43.807Z). === 2025-09-28 === * 08:54 [[gitlab:pexerik|@pexerik]] was approved. === 2025-09-27 === * 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved. === 2025-09-26 === * 16:57 "algorithmic" was rejected (pending since 2025-06-27T16:56:17.480Z). * 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved. * 13:12 [[gitlab:spushpit|@spushpit]] was approved. === 2025-09-20 === * 14:06 "bwiki" was rejected (pending since 2025-06-21T13:59:14.749Z). === 2025-09-16 === * 05:39 [[gitlab:deepchirp|@deepchirp]] was approved. === 2025-09-15 === * 22:00 [[gitlab:noisk8|@noisk8]] was approved. * 11:03 "ahonc" was rejected (pending since 2025-06-16T11:00:54.843Z). === 2025-09-13 === * 18:24 "a-ssh22" was rejected (pending since 2025-06-14T18:23:33.937Z). * 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved. * 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved. === 2025-09-12 === * 17:12 [[gitlab:suyash23|@suyash23]] was approved. * 00:46 "remotetravel" was rejected (pending since 2025-06-13T00:44:08.171Z). === 2025-09-10 === * 21:09 "jancborchardt" was rejected (pending since 2025-06-11T21:06:30.759Z). === 2025-09-09 === * 17:03 [[gitlab:vwf|@vwf]] was approved. * 06:36 [[gitlab:cactusisme|@cactusisme]] was approved. === 2025-09-08 === * 18:09 "birushandegeya" was rejected (pending since 2025-06-09T18:08:00.087Z). * 16:27 "ngarnsworthy" was rejected (pending since 2025-06-09T16:24:37.213Z). * 12:33 "zolgoyo" was rejected (pending since 2025-06-09T12:31:34.199Z). === 2025-09-06 === * 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved. === 2025-09-05 === * 21:45 [[gitlab:sakshi2|@sakshi2]] was approved. * 20:42 "abdukhaliq1" was rejected (pending since 2025-06-06T20:40:42.023Z). * 14:27 "beubsamy" was rejected (pending since 2025-06-06T14:27:06.781Z). === 2025-09-04 === * 23:27 "sdhehua" was rejected (pending since 2025-06-05T23:24:45.777Z). * 19:00 [[gitlab:perry|@perry]] was approved. * 11:24 "saintwolf" was rejected (pending since 2025-06-05T11:21:20.176Z). === 2025-09-02 === * 05:48 [[gitlab:aliu|@aliu]] was approved. === 2025-08-29 === * 13:30 "kksurendran066" was rejected (pending since 2025-05-30T13:27:48.755Z). === 2025-08-28 === * 22:18 "tauraamuix" was rejected (pending since 2025-05-29T22:16:08.228Z). === 2025-08-26 === * 19:03 [[gitlab:dikkulah|@dikkulah]] was approved. === 2025-08-22 === * 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved. === 2025-08-21 === * 07:39 [[gitlab:yuka|@yuka]] was approved. === 2025-08-19 === * 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved. === 2025-08-17 === * 14:27 "madhan13k" was rejected (pending since 2025-05-18T14:26:08.973Z). === 2025-08-15 === * 10:15 "mohammed_abukhadra" was rejected (pending since 2025-05-16T10:14:48.403Z). === 2025-08-11 === * 11:48 "hmmyesbro" was rejected (pending since 2025-05-12T11:45:24.350Z). === 2025-08-10 === * 13:15 [[gitlab:dactyl|@dactyl]] was approved. === 2025-08-09 === * 04:39 "xxxx100000" was rejected (pending since 2025-05-10T04:37:44.949Z). === 2025-08-08 === * 14:33 [[gitlab:josefanthony|@josefanthony]] was approved. === 2025-08-07 === * 23:42 [[gitlab:robins7|@robins7]] was approved. * 21:42 [[gitlab:pols12|@pols12]] was approved. * 17:15 "sbronson" was rejected (pending since 2025-05-08T17:15:08.834Z). * 14:57 [[gitlab:alvindulle|@alvindulle]] was approved. * 14:45 [[gitlab:xentos|@xentos]] was approved. * 06:27 "jamesboste" was rejected (pending since 2025-05-08T06:25:14.793Z). * 03:57 "ysun" was rejected (pending since 2025-05-08T03:55:07.348Z). === 2025-08-06 === * 21:51 "pols12" was rejected (pending since 2025-05-07T21:49:13.598Z). * 01:51 "okeamah" was rejected (pending since 2025-05-07T01:48:50.114Z). === 2025-08-05 === * 09:15 "mobashir-2013" was rejected (pending since 2025-05-06T09:14:24.069Z). === 2025-08-01 === * 08:00 "douginamug" was rejected (pending since 2025-05-02T07:57:38.317Z). === 2025-07-31 === * 02:30 [[gitlab:ads|@ads]] was approved. === 2025-07-27 === * 13:15 "mrico2703" was rejected (pending since 2025-04-27T13:13:12.346Z). * 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved. * 10:17 [[gitlab:fuzzew|@fuzzew]] was approved. * 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved. * 05:48 [[gitlab:ecoholic|@ecoholic]] was approved. === 2025-07-26 === * 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved. * 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved. * 11:48 [[gitlab:hridyakk|@hridyakk]] was approved. * 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved. * 11:45 [[gitlab:binetaa|@binetaa]] was approved. * 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved. * 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved. * 10:21 [[gitlab:flykrth|@flykrth]] was approved. * 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved. * 10:21 [[gitlab:aadhi13|@aadhi13]] was approved. * 10:21 [[gitlab:malavikam05|@malavikam05]] was approved. * 10:18 [[gitlab:nf609|@nf609]] was approved. * 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved. * 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved. === 2025-07-25 === * 09:49 [[gitlab:kasyap9|@kasyap9]] was approved. * 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved. === 2025-07-24 === * 19:36 [[gitlab:madutgn|@madutgn]] was approved. === 2025-07-23 === * 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved. === 2025-07-22 === * 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved. === 2025-07-21 === * 17:30 [[gitlab:asadiqui|@asadiqui]] was approved. * 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved. * 04:27 [[gitlab:damian|@damian]] was approved. === 2025-07-20 === * 09:42 "mike-khoroshun" was rejected (pending since 2025-04-20T09:42:22.732Z). === 2025-07-17 === * 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved. * 13:45 [[gitlab:envlh|@envlh]] was approved. === 2025-07-14 === * 10:24 [[gitlab:missguru|@missguru]] was approved. * 00:57 "clarfonthey" was rejected (pending since 2025-04-14T00:56:32.626Z). === 2025-07-13 === * 01:01 [[gitlab:l235|@l235]] was approved. === 2025-07-11 === * 03:06 "rodavlas" was rejected (pending since 2025-04-11T03:05:45.590Z). === 2025-07-06 === * 00:09 "lakasa" was rejected (pending since 2025-04-06T00:06:28.469Z). === 2025-07-05 === * 21:54 "ctrlzvi" was rejected (pending since 2025-04-05T21:54:12.542Z). * 14:30 "aminualiyu" was rejected (pending since 2025-04-05T14:27:22.617Z). === 2025-07-04 === * 03:15 [[gitlab:galstar|@galstar]] was approved. === 2025-07-02 === * 11:27 "vicolas11" was rejected (pending since 2025-04-02T11:25:12.682Z). === 2025-06-29 === * 23:12 "naomi723" was rejected (pending since 2025-03-30T23:09:24.630Z). === 2025-06-28 === * 16:21 "mudeh2372" was rejected (pending since 2025-03-29T16:18:27.057Z). === 2025-06-27 === * 23:18 "rony143" was rejected (pending since 2025-03-28T23:16:13.671Z). * 22:21 [[gitlab:rluts|@rluts]] was approved. === 2025-06-26 === * 13:54 "creativegurus" was rejected (pending since 2025-03-27T13:52:41.706Z). === 2025-06-24 === * 17:42 [[gitlab:devjadiya|@devjadiya]] was approved. * 14:00 "dominic-r" was rejected (pending since 2025-03-25T14:00:07.307Z). === 2025-06-21 === * 00:48 [[gitlab:vriaa|@vriaa]] was approved. === 2025-06-18 === * 15:21 "ayushkhati1" was rejected (pending since 2025-03-19T15:18:50.062Z). === 2025-06-17 === * 20:45 "chiomavero" was rejected (pending since 2025-03-18T20:44:13.967Z). * 00:27 [[gitlab:eggroll97|@eggroll97]] was approved. === 2025-06-14 === * 20:57 "volvox" was rejected (pending since 2025-03-15T20:56:34.018Z). === 2025-06-13 === * 16:09 [[gitlab:supergrey|@supergrey]] was approved. * 11:03 "chqaz" was rejected (pending since 2025-03-14T11:01:09.600Z). * 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved. * 10:15 "hearvox" was rejected (pending since 2025-03-14T10:13:13.112Z). === 2025-06-12 === * 15:18 "jlam" was rejected (pending since 2025-03-13T15:17:54.099Z). === 2025-06-09 === * 20:48 "dipanjansengupta" was rejected (pending since 2025-03-10T20:48:03.545Z). * 19:27 [[gitlab:reggycelly|@reggycelly]] was approved. * 14:51 "arendpieter" was rejected (pending since 2025-03-10T14:51:01.445Z). * 13:21 [[gitlab:greenreaper|@greenreaper]] was approved. * 09:33 [[gitlab:mmta|@mmta]] was approved. * 08:03 "a-ssh22" was rejected (pending since 2025-03-10T08:03:08.111Z). === 2025-06-08 === * 21:06 "mm-episodenlistedlvaupdater" was rejected (pending since 2025-03-09T21:04:06.323Z). === 2025-06-06 === * 11:06 [[gitlab:olea|@olea]] was approved. === 2025-06-05 === * 20:33 [[gitlab:encodedwp|@encodedwp]] was approved. * 15:00 [[gitlab:toluayo|@toluayo]] was approved. * 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved. * 11:54 "sdhehua" was rejected (pending since 2025-03-06T11:51:48.241Z). === 2025-06-03 === * 21:27 [[gitlab:wewakey|@wewakey]] was approved. * 12:36 "hunsimon2" was rejected (pending since 2025-03-04T12:34:56.520Z). * 11:54 "hunsimon" was rejected (pending since 2025-03-04T11:53:54.652Z). === 2025-06-02 === * 12:01 [[gitlab:jaimedes|@jaimedes]] was approved. === 2025-05-30 === * 18:00 "sathvik9105" was rejected (pending since 2025-02-28T17:59:42.867Z). * 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved. * 10:06 [[gitlab:gpsleo|@gpsleo]] was approved. === 2025-05-29 === * 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved. === 2025-05-28 === * 02:57 [[gitlab:saper|@saper]] was approved. === 2025-05-27 === * 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved. * 15:33 "satanluimm" was rejected (pending since 2025-02-25T15:32:48.101Z). === 2025-05-26 === * 23:57 "seyedali220" was rejected (pending since 2025-02-24T23:56:17.621Z). === 2025-05-21 === * 11:12 [[gitlab:guilherme|@guilherme]] was approved. === 2025-05-19 === * 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved. === 2025-05-18 === * 00:00 "xidme" was rejected (pending since 2025-02-15T23:58:56.796Z). === 2025-05-17 === * 02:39 "kdh8219" was rejected (pending since 2025-02-15T02:36:32.237Z). === 2025-05-16 === * 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved. === 2025-05-15 === * 04:30 "inspectorzer0" was rejected (pending since 2025-02-13T04:27:33.179Z). === 2025-05-14 === * 17:42 [[gitlab:llugo|@llugo]] was approved. === 2025-05-13 === * 20:18 "mmta" was rejected (pending since 2025-02-11T20:17:23.407Z). === 2025-05-11 === * 20:51 "jad" was rejected (pending since 2025-02-09T20:49:07.333Z). * 17:54 "nishchalsundan" was rejected (pending since 2025-02-09T17:52:25.761Z). * 16:39 "mohammed_abukhadra" was rejected (pending since 2025-02-09T16:39:03.730Z). === 2025-05-09 === * 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved. === 2025-05-08 === * 08:18 [[gitlab:mengeditch|@mengeditch]] was approved. === 2025-05-07 === * 03:45 "xluffy" was rejected (pending since 2025-02-05T03:45:14.181Z). === 2025-05-06 === * 16:54 "punhaniabhishek" was rejected (pending since 2025-02-04T16:53:50.758Z). * 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved. === 2025-05-02 === * 12:24 [[gitlab:tohaomg|@tohaomg]] was approved. * 11:48 [[gitlab:mavrikant|@mavrikant]] was approved. * 11:45 [[gitlab:daanvr|@daanvr]] was approved. === 2025-05-01 === * 09:09 "mjoerg" was rejected (pending since 2025-01-30T09:09:04.204Z). === 2025-04-30 === * 23:06 "sanskardubey" was rejected (pending since 2025-01-29T23:03:25.489Z). === 2025-04-29 === * 16:00 "geyslein" was rejected (pending since 2025-01-28T16:00:01.510Z). === 2025-04-26 === * 09:30 "anjali9027" was rejected (pending since 2025-01-25T09:28:07.064Z). === 2025-04-25 === * 18:00 "salahhazaa" was rejected (pending since 2025-01-24T17:58:30.030Z). * 15:15 [[gitlab:yiming|@yiming]] was approved. * 02:06 "mrchanmp" was rejected (pending since 2025-01-24T02:03:58.308Z). === 2025-04-23 === * 17:03 "rj2904" was rejected (pending since 2025-01-22T17:03:11.207Z). * 14:21 "nischay33" was rejected (pending since 2025-01-22T14:19:21.081Z). === 2025-04-22 === * 19:27 "dj80" was rejected (pending since 2025-01-21T19:25:28.498Z). * 14:30 [[gitlab:kaimamin|@kaimamin]] was approved. * 09:57 "debo" was rejected (pending since 2025-01-21T09:54:47.955Z). === 2025-04-21 === * 12:24 "unshell" was rejected (pending since 2025-01-20T12:21:59.686Z). === 2025-04-18 === * 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved. === 2025-04-16 === * 03:09 "dewey" was rejected (pending since 2025-01-15T03:06:17.488Z). === 2025-04-15 === * 19:45 "emdadul" was rejected (pending since 2025-01-14T19:42:29.285Z). === 2025-04-14 === * 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved. === 2025-04-11 === * 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved. === 2025-04-10 === * 04:12 "bhai420" was rejected (pending since 2025-01-09T04:10:29.430Z). === 2025-04-09 === * 05:03 "austinvarshney" was rejected (pending since 2025-01-08T05:02:34.175Z). === 2025-04-06 === * 15:36 [[gitlab:elph|@elph]] was approved. === 2025-04-02 === * 10:33 [[gitlab:ozge|@ozge]] was approved. === 2025-03-31 === * 20:15 "demandkey" was rejected (pending since 2024-12-30T20:14:23.096Z). * 15:18 [[gitlab:danyya|@danyya]] was approved. === 2025-03-28 === * 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved. * 15:54 [[gitlab:ilanen1|@ilanen1]] was approved. === 2025-03-25 === * 19:27 [[gitlab:irfo|@irfo]] was approved. * 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved. * 04:33 [[gitlab:paul26|@paul26]] was approved. * 04:18 "as1100k" was rejected (pending since 2024-12-24T04:18:06.813Z). === 2025-03-24 === * 11:33 "amzadkhankk" was rejected (pending since 2024-12-23T11:33:14.176Z). === 2025-03-23 === * 12:24 "wolfdo" was rejected (pending since 2024-12-22T12:23:35.056Z). === 2025-03-22 === * 09:45 [[gitlab:fjmustak|@fjmustak]] was approved. === 2025-03-20 === * 18:42 "sathishkokila" was rejected (pending since 2024-12-19T18:39:35.161Z). * 17:03 [[gitlab:alien4444|@alien4444]] was approved. * 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved. === 2025-03-19 === * 22:57 [[gitlab:r1f4t|@r1f4t]] was approved. * 19:03 "daniel24ps" was rejected (pending since 2024-12-18T19:00:21.249Z). * 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved. === 2025-03-18 === * 17:48 "rahulkundu1209" was rejected (pending since 2024-12-17T17:46:41.936Z). * 08:15 "kirtisikka972" was rejected (pending since 2024-12-17T08:13:25.487Z). === 2025-03-15 === * 13:30 "tulspal_sidhu" was rejected (pending since 2024-12-14T13:29:10.606Z). * 01:39 "peacedeadc" was rejected (pending since 2024-12-14T01:37:36.579Z). === 2025-03-14 === * 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved. * 02:33 "yxngtrtxll" was rejected (pending since 2024-12-13T02:31:51.658Z). === 2025-03-13 === * 14:36 [[gitlab:iccander|@iccander]] was approved. === 2025-03-12 === * 23:21 "jokerchic36" was rejected (pending since 2024-12-11T23:21:00.670Z). * 15:30 [[gitlab:naomi|@naomi]] was approved. * 15:27 [[gitlab:cobi|@cobi]] was approved. === 2025-03-11 === * 12:42 "mohitvermaxx" was rejected (pending since 2024-12-10T12:40:56.967Z). === 2025-03-10 === * 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved. === 2025-03-09 === * 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved. * 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved. === 2025-03-07 === * 16:54 [[gitlab:hswan|@hswan]] was approved. * 14:42 [[gitlab:atitkov|@atitkov]] was approved. * 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved. === 2025-03-06 === * 17:21 "johnmann" was rejected (pending since 2024-12-05T17:19:24.995Z). === 2025-03-05 === * 07:33 [[gitlab:monx9494|@monx9494]] was approved. === 2025-03-02 === * 21:21 "paul26" was rejected (pending since 2024-12-01T21:20:19.681Z). === 2025-03-01 === * 19:15 [[gitlab:izno|@izno]] was approved. * 12:45 [[gitlab:nyerho|@nyerho]] was approved. === 2025-02-28 === * 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved. * 13:09 "ashwinpraveengo" was rejected (pending since 2024-11-29T13:07:47.240Z). * 00:18 "eduardoaugusto" was rejected (pending since 2024-11-29T00:17:43.372Z). === 2025-02-27 === * 20:39 "volkanurl" was rejected (pending since 2024-11-28T20:37:18.101Z). === 2025-02-24 === * 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved. * 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved. * 19:06 [[gitlab:dhardy|@dhardy]] was approved. === 2025-02-22 === * 19:27 [[gitlab:owuh|@owuh]] was approved. === 2025-02-19 === * 16:06 [[gitlab:artemkloko|@artemkloko]] was approved. * 13:03 [[gitlab:jgafnea|@jgafnea]] was approved. === 2025-02-17 === * 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved. === 2025-02-16 === * 19:12 "gaurigupta21" was rejected (pending since 2024-11-17T19:11:07.416Z). === 2025-02-15 === * 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved. === 2025-02-14 === * 15:21 "nathanbnm" was rejected (pending since 2024-11-15T15:18:19.632Z). === 2025-02-13 === * 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved. * 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved. === 2025-02-12 === * 23:21 "monkeypatch999" was rejected (pending since 2024-11-13T23:20:38.398Z). * 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved. === 2025-02-11 === * 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved. === 2025-02-09 === * 16:15 "mohammed_abukhadra" was rejected (pending since 2024-11-10T16:15:18.361Z). === 2025-02-07 === * 21:33 "brennan" was rejected (pending since 2024-11-08T21:31:07.351Z). === 2025-02-06 === * 08:24 "mmta" was rejected (pending since 2024-11-07T08:22:36.724Z). * 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved. === 2025-02-05 === * 22:39 "chrissteinchen" was rejected (pending since 2024-11-06T22:38:16.673Z). === 2025-02-03 === * 07:45 "edriiic" was rejected (pending since 2024-11-04T07:44:46.849Z). * 01:12 "geppy" was rejected (pending since 2024-11-04T01:10:48.710Z). === 2025-02-02 === * 13:18 "funa-enpitu" was rejected (pending since 2024-11-03T13:15:46.065Z). === 2025-01-31 === * 23:42 "nfontes" was rejected (pending since 2024-11-01T23:39:41.755Z). * 22:51 "sbronson" was rejected (pending since 2024-11-01T22:50:31.871Z). * 00:42 [[gitlab:farid|@farid]] was approved. === 2025-01-27 === * 08:15 [[gitlab:eliza189|@eliza189]] was approved. === 2025-01-25 === * 09:51 [[gitlab:pamputt|@pamputt]] was approved. === 2025-01-23 === * 14:30 [[gitlab:lubianat|@lubianat]] was approved. * 11:45 [[gitlab:bootsa|@bootsa]] was approved. === 2025-01-21 === * 05:09 "niko" was rejected (pending since 2024-07-21T16:10:01.377Z). * 05:09 "thawizkid369777" was rejected (pending since 2024-07-18T17:42:44.493Z). * 05:09 "sarthaksingh2" was rejected (pending since 2024-07-10T11:31:30.470Z). * 05:09 "shriyakt" was rejected (pending since 2024-07-06T04:54:10.248Z). * 05:09 "akshaya" was rejected (pending since 2024-07-06T04:04:51.488Z). * 05:09 "alaka03aj" was rejected (pending since 2024-07-05T18:01:54.876Z). * 05:09 "sulochanaviji-5049" was rejected (pending since 2024-07-01T05:58:00.427Z). * 05:09 "nayanjnath" was rejected (pending since 2024-07-01T02:51:57.405Z). * 05:09 "sd44" was rejected (pending since 2024-06-30T04:28:51.436Z). * 05:09 "metavalent" was rejected (pending since 2024-06-29T01:37:14.210Z). * 05:09 "wicloudx" was rejected (pending since 2024-06-28T11:51:23.335Z). * 05:09 "debo" was rejected (pending since 2024-06-28T01:44:59.845Z). * 05:09 "bwiki" was rejected (pending since 2024-06-23T14:15:38.032Z). * 05:09 "toprak" was rejected (pending since 2024-06-23T11:35:50.819Z). * 05:09 "iristeller" was rejected (pending since 2024-06-14T20:53:48.959Z). * 05:09 "jcolvin" was rejected (pending since 2024-06-12T17:29:01.238Z). * 05:09 "kalyan" was rejected (pending since 2024-06-07T07:52:46.993Z). * 05:09 "bluecrystal" was rejected (pending since 2024-06-06T19:16:20.107Z). * 05:09 "iftttrohit" was rejected (pending since 2024-06-04T12:08:50.818Z). * 05:09 "pogpotato" was rejected (pending since 2024-06-03T17:58:21.684Z). * 05:09 "cptlausebaer" was rejected (pending since 2024-05-31T18:53:27.692Z). * 05:09 "hdevine825" was rejected (pending since 2024-05-31T17:04:18.279Z). * 05:09 "anaghaa18" was rejected (pending since 2024-05-25T19:14:31.803Z). * 05:09 "atharvanair04" was rejected (pending since 2024-05-25T14:24:52.825Z). * 05:09 "anasvemmully" was rejected (pending since 2024-05-25T06:10:27.261Z). * 05:09 "abhinavmohandas" was rejected (pending since 2024-05-25T06:05:24.825Z). * 05:09 "kksurendran06" was rejected (pending since 2024-05-25T06:04:38.082Z). * 05:09 "albertmarshall8896" was rejected (pending since 2024-05-23T09:32:05.462Z). * 05:09 "akellison" was rejected (pending since 2024-05-17T02:07:24.229Z). * 05:09 "mainowill" was rejected (pending since 2024-04-16T23:30:33.881Z). * 05:09 "bzhqc" was rejected (pending since 2024-04-16T19:50:38.676Z). * 05:09 "safan41" was rejected (pending since 2024-04-16T03:34:48.942Z). * 05:09 "mgagat" was rejected (pending since 2024-04-16T03:21:51.764Z). * 05:09 "okeamah" was rejected (pending since 2024-04-16T02:49:00.143Z). * 05:09 "xuhao61" was rejected (pending since 2024-04-15T23:45:09.083Z). * 04:47 "cybel" was rejected (pending since 2024-04-15T06:46:35.791Z). === 2025-01-20 === * 14:33 [[gitlab:your1|@your1]] was approved. === 2025-01-18 === * 10:09 [[gitlab:galrach600|@galrach600]] was approved. * 02:51 [[gitlab:blankeclair|@blankeclair]] was approved. === 2025-01-17 === * 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved. === 2025-01-15 === * 17:12 [[gitlab:smartse|@smartse]] was approved. === 2025-01-14 === * 17:03 [[gitlab:naorleizer|@naorleizer]] was approved. === 2025-01-13 === * 02:45 [[gitlab:wolf20482|@wolf20482]] was approved. === 2025-01-12 === * 17:45 [[gitlab:tamzin|@tamzin]] was approved. === 2025-01-11 === * 15:24 [[gitlab:bargioni|@bargioni]] was approved. * 14:30 [[gitlab:salelya|@salelya]] was approved. * 10:15 [[gitlab:malakatshy|@malakatshy]] was approved. * 05:21 [[gitlab:newmcpee|@newmcpee]] was approved. === 2025-01-09 === * 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved. === 2025-01-08 === * 16:21 [[gitlab:ukrface|@ukrface]] was approved. === 2024-12-28 === * 03:27 [[gitlab:twonum|@twonum]] was approved. === 2024-12-25 === * 06:09 [[gitlab:harsv567|@harsv567]] was approved. === 2024-12-21 === * 11:24 [[gitlab:amutha2002|@amutha2002]] was approved. === 2024-12-20 === * 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved. * 10:00 [[gitlab:ro-shines|@ro-shines]] was approved. * 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved. === 2024-12-18 === * 14:45 [[gitlab:soylacarli|@soylacarli]] was approved. === 2024-12-16 === * 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved. === 2024-12-15 === * 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved. === 2024-12-13 === * 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved. === 2024-12-10 === * 06:39 [[gitlab:ginaan|@ginaan]] was approved. === 2024-12-09 === * 05:45 [[gitlab:kallinavya|@kallinavya]] was approved. * 00:54 [[gitlab:viserion-7|@viserion-7]] was approved. === 2024-12-08 === * 17:27 [[gitlab:wargo|@wargo]] was approved. === 2024-12-05 === * 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved. === 2024-12-02 === * 21:21 [[gitlab:a930913|@a930913]] was approved. === 2024-12-01 === * 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved. === 2024-11-21 === * 13:45 [[gitlab:sascha|@sascha]] was approved. === 2024-11-19 === * 16:36 [[gitlab:jly|@jly]] was approved. === 2024-11-15 === * 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved. === 2024-11-14 === * 14:15 [[gitlab:stimoroll|@stimoroll]] was approved. === 2024-11-09 === * 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved. === 2024-11-07 === * 19:15 [[gitlab:zulf|@zulf]] was approved. * 05:33 [[gitlab:hassanamin|@hassanamin]] was approved. === 2024-11-06 === * 19:39 [[gitlab:daniuu|@daniuu]] was approved. * 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved. === 2024-10-09 === * 14:45 [[gitlab:jtweed|@jtweed]] was approved. * 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved. * 09:06 [[gitlab:wikibayer|@wikibayer]] was approved. === 2024-10-06 === * 10:27 [[gitlab:keerthan16|@keerthan16]] was approved. === 2024-10-04 === * 07:45 [[gitlab:hakimi97|@hakimi97]] was approved. === 2024-09-30 === * 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved. === 2024-09-28 === * 17:30 [[gitlab:webrunner95|@webrunner95]] was approved. === 2024-09-18 === * 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved. === 2024-09-14 === * 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved. === 2024-09-06 === * 08:48 [[gitlab:mickabarber|@mickabarber]] was approved. === 2024-08-27 === * 17:36 [[gitlab:edgars|@edgars]] was approved. === 2024-08-22 === * 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved. === 2024-08-14 === * 19:21 [[gitlab:jfk|@jfk]] was approved. === 2024-08-13 === * 17:57 [[gitlab:daxserver|@daxserver]] was approved. === 2024-08-11 === * 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved. === 2024-08-10 === * 08:42 [[gitlab:ashig|@ashig]] was approved. === 2024-08-09 === * 14:09 [[gitlab:masssly|@masssly]] was approved. === 2024-08-05 === * 22:15 [[gitlab:mrtortue|@mrtortue]] was approved. === 2024-08-02 === * 16:21 [[gitlab:dsantini|@dsantini]] was approved. === 2024-07-31 === * 11:54 [[gitlab:cptviraj|@cptviraj]] was approved. === 2024-07-30 === * 19:09 [[gitlab:iniquity|@iniquity]] was approved. * 10:00 [[gitlab:collins|@collins]] was approved. === 2024-07-27 === * 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved. === 2024-07-25 === * 12:36 [[gitlab:mszabo|@mszabo]] was approved. * 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved. === 2024-07-24 === * 08:05 [[gitlab:dragoniez|@dragoniez]] was approved. === 2024-07-23 === * 06:54 [[gitlab:mirji|@mirji]] was approved. === 2024-07-16 === * 10:00 [[gitlab:lakejason0|@lakejason0]] was approved. === 2024-07-12 === * 11:33 [[gitlab:cn|@cn]] was approved. * 08:12 [[gitlab:unchampignon|@unchampignon]] was approved. === 2024-07-07 === * 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved. * 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved. === 2024-07-06 === * 11:18 [[gitlab:dibya|@dibya]] was approved. * 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved. === 2024-07-05 === * 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved. === 2024-07-02 === * 19:00 [[gitlab:ebrahim|@ebrahim]] was approved. === 2024-07-01 === * 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved. * 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved. === 2024-06-30 === * 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved. * 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved. * 06:30 [[gitlab:rihaan180|@rihaan180]] was approved. === 2024-06-27 === * 17:36 [[gitlab:driedmueller|@driedmueller]] was approved. === 2024-06-19 === * 12:57 [[gitlab:audreypenven|@audreypenven]] was approved. === 2024-06-16 === * 01:18 [[gitlab:roysmith|@roysmith]] was approved. === 2024-06-08 === * 02:45 [[gitlab:jleedev|@jleedev]] was approved. === 2024-06-03 === * 13:57 [[gitlab:afeder|@afeder]] was approved. === 2024-06-01 === * 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved. === 2024-05-30 === * 16:42 [[gitlab:krlsca|@krlsca]] was approved. === 2024-05-28 === * 11:24 [[gitlab:rickijay|@rickijay]] was approved. === 2024-05-26 === * 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved. === 2024-05-25 === * 07:24 [[gitlab:jony|@jony]] was approved. === 2024-05-23 === * 08:45 [[gitlab:lepticed7|@lepticed7]] was approved. === 2024-05-22 === * 20:42 [[gitlab:echecs|@echecs]] was approved. === 2024-05-21 === * 13:33 [[gitlab:mbs|@mbs]] was approved. === 2024-05-19 === * 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved. === 2024-05-18 === * 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved. === 2024-05-17 === * 08:54 [[gitlab:grapedog|@grapedog]] was approved. === 2024-05-08 === * 19:42 [[gitlab:kelhurd|@kelhurd]] was approved. * 19:06 [[gitlab:khurd|@khurd]] was approved. === 2024-05-06 === * 19:48 [[gitlab:j3j5|@j3j5]] was approved. * 12:06 [[gitlab:tk-999|@tk-999]] was approved. === 2024-05-05 === * 22:09 [[gitlab:pppery|@pppery]] was approved. * 20:33 [[gitlab:sakretsu|@sakretsu]] was approved. * 12:12 [[gitlab:waterquark|@waterquark]] was approved. === 2024-05-04 === * 09:03 [[gitlab:multichill|@multichill]] was approved. * 07:42 [[gitlab:abaris|@abaris]] was approved. === 2024-05-03 === * 14:57 [[gitlab:maurusian|@maurusian]] was approved. === 2024-04-24 === * 05:48 [[gitlab:wolfinux|@wolfinux]] was approved. === 2024-04-23 === * 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved. === 2024-04-21 === * 06:51 [[gitlab:alon|@alon]] was approved. === 2024-04-17 === * 23:33 [[gitlab:derenrich|@derenrich]] was approved. === 2024-04-16 === * 17:18 [[gitlab:valcio|@valcio]] was approved. === 2024-04-14 === * 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved. === 2024-04-06 === * 12:48 [[gitlab:theprotonade|@theprotonade]] was approved. === 2024-04-02 === * 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved. === 2024-03-30 === * 13:36 [[gitlab:lpintscher|@lpintscher]] was approved. === 2024-03-26 === * 17:09 [[gitlab:eenabulele|@eenabulele]] was approved. === 2024-03-25 === * 14:27 [[gitlab:tuukka|@tuukka]] was approved. === 2024-03-24 === * 12:24 [[gitlab:firefly|@firefly]] was approved. === 2024-03-21 === * 19:33 [[gitlab:universal-omega|@universal-omega]] was approved. === 2024-03-17 === * 10:36 [[gitlab:bisel91|@bisel91]] was approved. === 2024-03-16 === * 10:09 [[gitlab:delord|@delord]] was approved. * 00:42 [[gitlab:athulvis1|@athulvis1]] was approved. === 2024-03-15 === * 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved. * 08:30 [[gitlab:peachey88|@peachey88]] was approved. * 06:51 [[gitlab:derick|@derick]] was approved. === 2024-03-12 === * 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved. === 2024-03-06 === * 13:21 [[gitlab:desianabae1|@desianabae1]] was approved. === 2024-03-05 === * 19:21 [[gitlab:ep1c|@ep1c]] was approved. * 16:33 [[gitlab:jasmine|@jasmine]] was approved. === 2024-03-02 === * 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved. === 2024-02-29 === * 23:18 [[gitlab:arandomname123|@arandomname123]] was approved. * 18:03 [[gitlab:baba|@baba]] was approved. * 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved. * 03:09 [[gitlab:sds|@sds]] was approved. === 2024-02-27 === * 23:33 [[gitlab:lofhi|@lofhi]] was approved. === 2024-02-15 === * 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved. === 2024-02-14 === * 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved. === 2024-02-13 === * 13:06 [[gitlab:dringsim|@dringsim]] was approved. === 2024-02-12 === * 17:36 [[gitlab:haak|@haak]] was approved. === 2024-02-05 === * 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved. * 17:14 [[gitlab:ahecht|@ahecht]] was approved. === 2024-02-01 === * 09:27 [[gitlab:arinaigum|@arinaigum]] was approved. * 00:15 [[gitlab:jas42|@jas42]] was approved. * 00:15 [[gitlab:edhu|@edhu]] was approved. * 00:15 [[gitlab:marnanel|@marnanel]] was approved. * 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved. * 00:15 [[gitlab:amasotti|@amasotti]] was approved. * 00:15 [[gitlab:deni|@deni]] was approved. * 00:15 [[gitlab:cyber|@cyber]] was approved. * 00:15 [[gitlab:saroj|@saroj]] was approved. === 2024-01-29 === * 21:42 [[gitlab:rgupta|@rgupta]] was approved. === 2024-01-07 === * 09:48 [[gitlab:lutrome|@lutrome]] was approved. === 2024-01-05 === * 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved. * 02:51 [[gitlab:braunobruno|@braunobruno]] was approved. * 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved. * 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved. === 2024-01-03 === * 14:45 [[gitlab:gabina|@gabina]] was approved. === 2024-01-02 === * 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved. === 2023-12-23 === * 00:33 [[gitlab:aram|@aram]] was approved. === 2023-12-22 === * 16:24 [[gitlab:elpitareio|@elpitareio]] was approved. === 2023-12-21 === * 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved. * 00:43 [[gitlab:ederporto|@ederporto]] was approved. * 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved. * 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved. * 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved. * 00:43 [[gitlab:air7538|@air7538]] was approved. * 00:43 [[gitlab:anzx|@anzx]] was approved. * 00:43 [[gitlab:tekask1903|@tekask1903]] was approved. * 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved. * 00:42 [[gitlab:mpaa|@mpaa]] was approved. * 00:42 [[gitlab:kutay|@kutay]] was approved. * 00:42 [[gitlab:wattmto|@wattmto]] was approved. c9h9pqupxdfc3yzd6mdc60fvobvumw1